diff --git a/.gitignore b/.gitignore index 64d1f4f..693fdf4 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,9 @@ dist/ # Node node_modules/ packages/web-app/dist/ +packages/web-app/test-results/ +packages/web-app/playwright-report/ +packages/web-app/playwright/.cache/ # Virtual environment .venv/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 73e694b..b0f846b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,52 +1,161 @@ # Contributing to DataLex -Thanks for contributing. This project includes a React web app, a Node API server, and a Python core/CLI. - -## Development Setup -1. Clone the repository and enter the project root. -2. Install Node dependencies: - - `npm --prefix packages/api-server install` - - `npm --prefix packages/web-app install` -3. Create Python venv and install requirements: - - `python3 -m venv .venv` - - `source .venv/bin/activate` - - `pip install -r requirements.txt` - -## Run Locally -- API: `npm --prefix packages/api-server run dev` -- Web: `npm --prefix packages/web-app run dev` -- CLI example: `./datalex validate model-examples/starter-commerce.model.yaml` - -## Branch and PR Flow +Thanks for contributing. This project is a monorepo with three pieces: + +- `packages/core_engine/` — Python loader, dialects, dbt integration, packages +- `packages/api-server/` — Node.js API the web UI talks to +- `packages/web-app/` — React/Vite studio (Zustand + React Flow) +- `packages/cli/` — `datalex` entry point + +## Development setup + +### Prerequisites + +- **Python 3.9+** with `pip` and `venv` +- **Node 20+** (`nvm use 20` if you use nvm) +- **Git** + +### One-time bootstrap + +```bash +git clone https://github.com/duckcode-ai/DataLex.git +cd DataLex +python3 -m venv .venv && source .venv/bin/activate +pip install -e '.[serve,duckdb]' # core_engine + CLI + connector +npm --prefix packages/api-server install +npm --prefix packages/web-app install +``` + +`pip install -e '.[serve,duckdb]'` installs `datalex_core` + the `datalex` +CLI in editable mode and pulls the bundled Node runtime used by +`datalex serve`. Add more connector extras as needed: +`'.[serve,postgres,snowflake,bigquery,databricks]'`. + +## Running locally + +Two supported modes: + +### Single-command (production-like) + +```bash +datalex serve --project-dir . +``` + +Serves the API and the pre-built web bundle together on +`http://localhost:3030`. Uses the portable Node that `[serve]` +installed. Good for smoke-testing a change in a real browser. + +### Hot-reload (for UI work) + +Two terminals: + +```bash +# Terminal 1 — API on :3006 +npm --prefix packages/api-server run dev + +# Terminal 2 — Vite dev server on :5173 with HMR +# (vite.config.js proxies /api → :3006 for you) +npm --prefix packages/web-app run dev +``` + +Open `http://localhost:5173`. The Vite proxy forwards every `/api/*` +call to the api-server, so the UI talks to the live backend while +HMR rebuilds React components on save. + +CLI during development (for package-level hacks): `./datalex `. + +## Testing + +### Python (core_engine + datalex) + +```bash +python3 -m unittest -v tests/test_mvp.py tests/test_cli_dx.py tests/test_policy_engine_v2.py +./datalex validate-all --schema schemas/model.schema.json +``` + +### API server + +```bash +npm --prefix packages/api-server test +``` + +### Web app — unit tests (fast, no browser) + +```bash +npm --prefix packages/web-app test +``` + +Runs everything in `packages/web-app/tests/*.test.js` via Node's +built-in test runner. + +### Web app — Playwright end-to-end (local-dev only) + +```bash +# One-time: install browsers +npx --prefix packages/web-app playwright install chromium + +# Clone + parse the jaffle-shop fixture once (needs dbt-duckdb) +cd packages/web-app/test-results/jaffle-shop # created by global-setup +pip install dbt-duckdb && dbt deps && dbt parse --profiles-dir . + +# Run the suite (starts api + vite via Playwright webServer) +npm --prefix packages/web-app run test:e2e +``` + +The E2E suite clones `https://github.com/dbt-labs/jaffle-shop` into +`packages/web-app/test-results/jaffle-shop/` on first run and reuses +the checkout afterwards. It drives the real user journey: import → +diagram → (with `E2E_FULL=1`) rename cascade → autosave → auto-commit +→ dry-run apply. + +**CI does not run this suite.** It requires dbt-core + a parsed +manifest on disk, which is too heavy and too flaky for every PR. The +backend contracts are already covered by api-server unit tests; the +Playwright suite is a local-dev regression tool for UI changes. +See [packages/web-app/e2e/README.md](packages/web-app/e2e/README.md). + +## Branch and PR flow + 1. Create a branch from `main`. 2. Keep changes focused and atomic. -3. Add or update tests for behavior changes. -4. Run relevant checks before opening PR. -5. Open PR with clear summary, impact, and test evidence. - -## Recommended Checks -- Python unit tests: - - `python3 -m unittest -v tests/test_mvp.py tests/test_cli_dx.py tests/test_policy_engine_v2.py` -- Web tests: - - `npm --prefix packages/web-app test` -- Validate example models: - - `./datalex validate-all --schema schemas/model.schema.json` - -## Commit Style -- Use short, imperative commit messages. -- Include scope when helpful, for example: `docs: update security policy`. - -## Coding Expectations -- Keep changes backward compatible unless a breaking change is explicitly discussed. +3. Add or update tests for behavior changes (unit **and** E2E when the + change affects the UI loop). +4. Run the relevant test suites locally before opening the PR. +5. Open the PR with a clear summary, user-visible impact, and test + evidence. + +CI runs on every PR: + +- `api-server-tests.yml` — `packages/api-server/` unit tests +- `model-quality.yml` — core_engine unit tests + policy checks +- `datalex.yml` — `datalex validate` / `diff` / `dbt emit` on touched + DataLex projects + +Playwright E2E is **not** in CI — run it locally before opening a PR +that changes import, canvas, or save-path behavior. + +## Commit style + +- Short, imperative commit messages. +- Include scope when helpful: `docs: update contributing guide`, + `web-app: drop bundled jaffle-shop fixture`. + +## Coding expectations + +- Keep changes backward compatible unless a breaking change is + explicitly discussed. - Update docs/examples when behavior or CLI output changes. - Avoid committing secrets, credentials, or local environment files. -## Reporting Bugs and Requesting Features +## Reporting bugs / requesting features + - Open a GitHub issue with reproduction steps and expected behavior. -- For connector issues, include connector type, redacted config, and failing command/log excerpt. +- For connector issues, include connector type, redacted config, and + the failing command/log excerpt. ## Cutting a release + See [RELEASING.md](RELEASING.md) for the full process. Short version: -bump `project.version` in `pyproject.toml`, move items from `[Unreleased]` -into a new dated section in `CHANGELOG.md`, merge, then push a signed -`vX.Y.Z` tag. CI publishes to PyPI automatically. +bump `project.version` in `pyproject.toml`, move items from +`[Unreleased]` into a new dated section in `CHANGELOG.md`, merge, then +push a signed `vX.Y.Z` tag. CI publishes to PyPI automatically. diff --git a/README.md b/README.md index 39beb8d..37a74fe 100644 --- a/README.md +++ b/README.md @@ -87,7 +87,7 @@ have in hand: | You have... | Tutorial | Time | |--------------------------------------------|--------------------------------------------------------------------|-------| -| Nothing — just want the demo | [Jaffle-shop one-click walkthrough](docs/tutorials/jaffle-shop-walkthrough.md) | 3 min | +| Nothing — want to try with a known-good dbt repo | [Walk through jaffle-shop end-to-end](docs/tutorials/jaffle-shop-walkthrough.md) | 5 min | | An existing dbt project (folder or git) | [Import an existing dbt project](docs/tutorials/import-existing-dbt.md) | 5 min | | A live warehouse (Snowflake/Postgres/…) | [Pull a warehouse schema](docs/tutorials/warehouse-pull.md) | 7 min | | CLI-only, no UI | [CLI dbt-sync tutorial](docs/tutorial-dbt-sync.md) | 5 min | @@ -242,7 +242,8 @@ dbt parse - **[Getting started](docs/getting-started.md)** — the one-page map covering install, the three GUI paths, and the mental model. - **[Jaffle-shop walkthrough](docs/tutorials/jaffle-shop-walkthrough.md)** — - 3-minute offline demo of every UI feature. + end-to-end demo: clone the real jaffle-shop repo, import it, rename an + entity, commit back to git. - **[Import an existing dbt project](docs/tutorials/import-existing-dbt.md)** — 5-minute bring-your-own-repo flow (local folder or git URL). - **[Pull a warehouse schema](docs/tutorials/warehouse-pull.md)** — diff --git a/docs/getting-started.md b/docs/getting-started.md index 3e5e18c..31652d3 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -28,7 +28,7 @@ pip install 'datalex-cli[serve,all]' # every driver + Node | You have... | Start here | Time | |-----------------------------------------------|----------------------------------------------------------------|-------| -| Nothing — just want the demo | [Scenario 1 — jaffle-shop demo](#scenario-1--jaffle-shop-demo) | 3 min | +| Nothing — want to try with a canonical dbt repo | [Scenario 1 — clone jaffle-shop](#scenario-1--clone-jaffle-shop) | 5 min | | An existing dbt project on disk | [Scenario 2 — your local dbt repo](#scenario-2--your-local-dbt-repo) | 5 min | | A dbt repo on GitHub you want to try | [Scenario 3 — a git URL](#scenario-3--a-git-url) | 4 min | | A live warehouse, no dbt yet | [Scenario 4 — warehouse pull](#scenario-4--live-warehouse-pull) | 7 min | @@ -36,23 +36,27 @@ pip install 'datalex-cli[serve,all]' # every driver + Node --- -## Scenario 1 — Jaffle-shop demo +## Scenario 1 — Clone jaffle-shop -The fastest way to see if DataLex fits how you think. No dbt repo -needed, no warehouse, fully offline. +The fastest way to see if DataLex fits how you think. Uses the real +`dbt-labs/jaffle-shop` repo — no bundled demo, no surprises when you +switch to your own project later. ```bash pip install 'datalex-cli[serve]' +git clone https://github.com/dbt-labs/jaffle-shop ~/src/jaffle-shop datalex serve ``` -Browser opens. Click **Import dbt repo → Load jaffle-shop demo**. The -Explorer fills with `models/staging/`, `models/marts/`, the canvas -shows an ER diagram with relationships, and the inspector renders -every column. - -Nothing is written to disk. Close the tab and everything is gone. -When you want the real workflow, go to Scenario 2. +In the UI: **Import dbt repo → Git URL tab** → paste +`https://github.com/dbt-labs/jaffle-shop` → **Import**. The API +server clones on your behalf, runs the importer, and shows the +**Import Results** panel. Click **Open project**. + +Prefer Save-All-writes-to-disk? Use the **Local folder** tab instead, +point it at the clone you just made (`~/src/jaffle-shop`), keep +**Edit in place** checked. Every UI edit then lands in the clone and +`git diff` shows normal dbt changes. 📖 **Full walkthrough:** [tutorials/jaffle-shop-walkthrough.md](tutorials/jaffle-shop-walkthrough.md) diff --git a/docs/tutorials/jaffle-shop-walkthrough.md b/docs/tutorials/jaffle-shop-walkthrough.md index 66b987c..fcdeca9 100644 --- a/docs/tutorials/jaffle-shop-walkthrough.md +++ b/docs/tutorials/jaffle-shop-walkthrough.md @@ -1,9 +1,8 @@ -# Jaffle-shop: the three-minute demo +# Jaffle-shop end-to-end walkthrough -The fastest way to see every DataLex feature without connecting a -warehouse or cloning a dbt repo. The jaffle-shop fixture is a trimmed -version of `dbt-labs/jaffle-shop` checked into the wheel, so this -entire flow works offline. +The fastest way to see every DataLex feature with a real, canonical +dbt project: clone `dbt-labs/jaffle-shop` from GitHub and drive the +full round-trip — import → diagram → edit → autosave → git. You'll end with: @@ -11,10 +10,11 @@ You'll end with: as both a file tree and an ER diagram - Inline lint warnings for every column missing `description`, `data_type`, or primary-key tests -- A feel for drag-to-relate, position persistence, and the - diff/history panels — the same UX you'll use for your real project +- A real `.git` history of your edits — DataLex writes back into the + cloned repo, so `git log` / `git diff` show normal dbt changes -**Time:** 3 minutes. **Prerequisites:** Python 3.9+ with pip. +**Time:** 5 minutes. **Prerequisites:** Python 3.9+, Git, and network +access to `github.com`. --- @@ -37,36 +37,62 @@ The first `datalex serve` call prints something like: A browser tab opens on `http://localhost:3030`. If it doesn't, open that URL manually or re-run with `--no-browser` and copy the link. -## Step 2 — Load the jaffle-shop demo - -1. In the top bar, click **Import dbt repo** (the folder icon with - "Dep" label, next to **Open project**). -2. The **Import dbt repository** dialog opens. At the top there's a - **Load jaffle-shop demo** button — click it. -3. The dialog closes and the Explorer (left panel) populates with the - full tree: - ``` - models/ - staging/ - stg_customers.yml - stg_orders.yml - stg_order_items.yml - stg_products.yml - stg_stores.yml - stg_supplies.yml - marts/ - customers.yml - orders.yml - order_items.yml - products.yml - sources/ - jaffle_shop_raw.yml - ``` - -The fixture is bundled as YAML files inside the wheel — no network -call, no git clone. If you ever re-run `dm dbt sync` against the real -`dbt-labs/jaffle-shop` repo you'll get the same tree (modulo the -`meta.datalex.dbt.*` timestamps). +## Step 2 — Import jaffle-shop from GitHub + +Two equivalent paths — pick whichever fits your workflow. + +### Option A — Let DataLex do the clone for you + +1. In the top bar, click **Import dbt repo** (the folder-with-arrow + icon). The **Import dbt repository** dialog opens on the **Git URL** + tab by default. +2. Paste `https://github.com/dbt-labs/jaffle-shop` into the **Git URL** + field. Leave the branch as `main`. Keep **Skip live warehouse + introspection** checked (we don't have warehouse creds yet). +3. Click **Import**. DataLex shells out to `git clone` on the API + server, runs `datalex dbt import` against the checkout, and shows + the **Import Results** panel. +4. Skim the report: how many models imported, the manifest-only banner + (no warehouse creds was fine), any columns with `type: unknown`, + and any unresolved relationships. Click **Open project**. + +This path is read-only by design — the import tree lives in memory so +you can explore without mutating anything on disk. Save All is +disabled. If you want edits to persist, use Option B. + +### Option B — Clone yourself, then open as a project + +```bash +git clone https://github.com/dbt-labs/jaffle-shop ~/src/jaffle-shop +``` + +1. Back in the UI, open **Import dbt repo** → **Local folder** tab. +2. Paste the absolute path, e.g. `/Users/you/src/jaffle-shop`. +3. Leave **Edit in place** checked (the default). This registers the + folder as a DataLex project: Save All writes edits back into each + model's original `.yml`, and `git diff` in the clone shows normal + dbt changes. +4. Click **Import**, review the Results panel, then **Open project**. + +Whichever path you pick, the Explorer (left panel) populates with the +full jaffle-shop tree: + +``` +models/ + staging/ + stg_customers.yml + stg_orders.yml + stg_order_items.yml + stg_products.yml + stg_supplies.yml + stg_locations.yml + marts/ + customers.yml + orders.yml + order_items.yml + products.yml + locations.yml +``` ## Step 3 — Build your first diagram @@ -86,8 +112,8 @@ which models to visualize together. `stg_customers.customer_id` renders automatically — inferred from the dbt `tests: - relationships:` on that column. 3. (Alternative) Drag `models/staging/stg_customers.yml` from the - Explorer onto the canvas. Each referenced model still renders as - an entity — the picker and drag-drop are interchangeable. + Explorer onto the canvas. Each model still renders as an entity — + the picker and drag-drop are interchangeable. 4. Reposition nodes by dragging. The positions land in the diagram YAML's `entities[].x/y` — not in the model files — so you can have a second diagram with different coordinates for the same models. @@ -102,80 +128,66 @@ Click `models/staging/stg_customers.yml` in the Explorer. - **Right panel** shows the Inspector: tabs for Columns, Relationships, Indexes, Enums, Tests. - **Columns tab** lists each column. Any column missing a - `description` or `data_type` shows a warning pill — that's the PR A - lint rule (`packages/web-app/src/lib/dbtLint.js`) running client-side + `description` or `data_type` shows a warning pill — that's the lint + rule (`packages/web-app/src/lib/dbtLint.js`) running client-side with no save-cost. Try renaming a column description: click the description cell, type -something, hit Enter. The YAML updates in-memory; the **Diff** panel -at the bottom shows the pending change as a red/green patch. - -## Step 5 — Drag to create a relationship - -On the canvas, each column has two tiny handles (left = target, -right = source). - -1. Drag from `stg_orders.customer_id` (right handle) to - `stg_customers.customer_id` (left handle). -2. A **New relationship** dialog opens, pre-filled with - `from: stg_orders.customer_id`, `to: stg_customers.customer_id`, - cardinality `many_to_one`. Give it a name like - `fk_orders_customers`, optionally mark it `identifying` or - `optional`, and hit **Create**. The dialog validates both - endpoints against the resolved model graph — if either column - doesn't exist, an inline error blocks submit (no silent write, no - toast). -3. A new FK edge renders. The Diff panel shows a new - `relationships:` block landed under `stg_orders`. - -## Step 6 — Move a node; confirm it sticks - -Drag `stg_customers` 300 px to the right. Reload the tab -(`⌘R` / `Ctrl+R`). The node stays where you put it because the canvas -wrote the new `x/y` into your active `.diagram.yaml`'s `entities[]` -list on `onNodeDragStop`. (When no diagram is active, positions fall -back to a `display:` block on the entity YAML itself.) - -## Step 7 — Try undo / redo - -Every mutating action (column edit, relationship add, position change) -pushes to a per-file history stack capped at 50 entries. - -- `⌘Z` reverts the last change -- `⌘⇧Z` re-applies - -The Chrome header's Undo and Redo buttons drive the same store — -they're live now, no longer decorative. - -## Step 8 — Validate + aggregate lint - -Click the **Validation** tab in the bottom panel. It aggregates every -lint warning and error across the whole tree, grouped by file. For -jaffle-shop you'll see a handful of "column missing description" -warnings — useful guide for a real import. - -If the active file has relationships whose endpoints reference a -missing entity or column, a red **Dangling relationships** banner -appears at the top of the panel, one card per finding, with a -**Remove dangling** button that rewrites the file and drops just the -offending entries. - -## Step 9 — Save the project to a real folder (optional) - -The jaffle-shop demo lives in-memory. If you want to write it to disk -for real git tracking: - -1. Click the **Save All** button in the top bar (the "All" download - icon — only enabled when you have a real project open). -2. Or use the File menu → New Project, pick a folder, then re-trigger - the import; the tree writes to your chosen directory. - -Once on disk: - -```bash -cd ~/my-jaffle-clone -git init && git add . && git commit -m "chore: jaffle-shop baseline" -``` +something, blur. The YAML updates in-memory and **autosave** flushes +the change to disk ~800ms later — you'll see the **Diff** panel at +the bottom transition from pending to clean. + +## Step 5 — Rename an entity and watch the cascade + +1. In the Explorer, right-click `stg_customers.yml` → **Rename + entity…** +2. Change the entity name from `stg_customers` to `stg_customer`. +3. Preview the rename. DataLex scans the whole project and lists + every file that will be rewritten — `stg_orders.yml`, + `customers.yml`, the diagram, and so on — each FK and relationship + ref updated atomically. +4. Click **Rename**. The server snapshots every target file, applies + the rewrites in a single transaction, and only then moves the file. + If any write fails, the whole thing rolls back. + +On Option B (edit-in-place), run `git diff` in the jaffle-shop clone — +you'll see the refactor as a coherent commit-sized change across +multiple files. + +## Step 6 — Turn on auto-commit (optional) + +1. Open the Commit dialog (`⌘⇧G` or the branch icon in the Chrome + header). +2. Enable **Auto-commit on save**. +3. Back in the inspector, change three field descriptions in quick + succession. Auto-commit debounces bursty saves: within ~3s you'll + see **exactly one** new commit in `git log`. + +Failure mode: if the commit fails (e.g. missing `user.email`), the +save itself still succeeds — the auto-commit error surfaces as a +toast so you can fix the config and retry manually. + +## Step 7 — Apply DDL to a warehouse (optional) + +1. In an open `.model.yaml`, press `⌘K` → **Apply to warehouse…** +2. Pick a dialect (DuckDB for a throwaway local run, Snowflake / BQ / + Databricks if you have a connector profile saved). +3. Click **Generate DDL** — the preview shows the forward-engineered + SQL. +4. Pick a connector profile. Leave **Dry run** checked for the first + pass; hit **Dry run**. The server compiles and validates against + the target without executing. +5. Uncheck **Dry run** → **Apply** when you're ready. + +The endpoint is gated by `DM_ENABLE_DIRECT_APPLY` on the server. When +disabled (the GitOps default), the dialog instead instructs you to +commit the generated SQL and deploy via CI/CD. + +## Step 8 — Export a PNG of the diagram + +With any diagram open, press `⌘⇧E`. A PNG of the current canvas +downloads. That same action lives in the diagram toolbar overflow +menu for discoverability. ## What to do next @@ -188,7 +200,8 @@ git init && git add . && git commit -m "chore: jaffle-shop baseline" | Symptom | Fix | |---------------------------------------------|--------------------------------------------------------------------| -| "Load jaffle-shop demo" button does nothing | Open devtools console. If you see a `glob()` error, the fixture wasn't bundled — reinstall with a recent wheel. | -| Explorer looks flat, no folders | You hit the single-file fallback — check the browser console for a `buildFileTree` error. | -| Node positions reset on reload | The YAML didn't save. Check for a red Save indicator in the header; save explicitly. | +| Git-URL import fails with a network error | Check that the API server has GitHub access (firewalls, proxies). Re-try with Option B: clone locally, then use the **Local folder** tab. | +| Import Results banner says "manifest-only" | Expected — jaffle-shop doesn't ship a `profiles.yml` wired to your machine. Column `data_type`s show `unknown` until you add warehouse creds. | +| Rename cascade complains about a file | The atomic endpoint rolls the whole rename back on any write failure. Fix the reported file (permissions, locks) and retry. | | Diff panel keeps showing changes after save | Stale editor state — hit `⌘R`. The in-flight Zustand store and the on-disk bytes should match. | +| Auto-commit produces no commit | Check `git config user.email` inside the cloned repo. The Chrome status bar shows the last auto-commit error as a toast. | diff --git a/packages/api-server/index.js b/packages/api-server/index.js index 9ef28ff..03f8ea6 100644 --- a/packages/api-server/index.js +++ b/packages/api-server/index.js @@ -3245,8 +3245,8 @@ app.post("/api/dbt/import", requireAdmin, express.json({ limit: "2mb" }), async // Walk outDir and build an in-memory tree of produced YAML files. Keeping the // response self-contained means the UI can ingest the result without any - // further disk access — important for the "Load jaffle-shop demo" flow where - // the user may not have chosen a project folder yet. + // further disk access — important for the git-URL import flow where the + // user may not have chosen a project folder yet. const tree = []; const walk = (dir, rel = "") => { let entries = []; diff --git a/packages/web-app/e2e/README.md b/packages/web-app/e2e/README.md new file mode 100644 index 0000000..8c47ab9 --- /dev/null +++ b/packages/web-app/e2e/README.md @@ -0,0 +1,64 @@ +# DataLex web-app E2E tests (local-dev only) + +Playwright end-to-end suite that exercises the real user journey against +a real cloned dbt project (by default [dbt-labs/jaffle-shop][jaffle]). +**This suite does not run in CI** — it needs dbt-core installed and a +parsed `target/manifest.json` on disk, which is too heavy and too flaky +for every PR. It stays in the tree as a local-dev tool for changes that +touch the import or canvas flow. + +[jaffle]: https://github.com/dbt-labs/jaffle-shop + +## What runs + +- `global-setup.js` clones jaffle-shop into + `packages/web-app/test-results/jaffle-shop/` once per machine, cached + on subsequent runs. +- Before you run the spec, **you must parse the dbt project once** so + `target/manifest.json` exists: + ```bash + cd packages/web-app/test-results/jaffle-shop + pip install dbt-duckdb + dbt deps + dbt parse --profiles-dir . # or point DBT_PROFILES_DIR at your own + ``` +- `critical-path.spec.js` drives the loop: import local folder → open + project → (with `E2E_FULL=1`) rename cascade, autosave, auto-commit, + Apply-to-Warehouse dry run. +- `import-api.spec.js` hits `/api/dbt/import` directly — no DOM — for a + cheap regression gate on the importer contract. + +## Running locally + +```bash +# From repo root, one-time: +npm --prefix packages/web-app install +npx --prefix packages/web-app playwright install chromium + +# Run the full suite (starts api + web via Playwright webServer): +npm --prefix packages/web-app run test:e2e + +# Interactive mode: +npm --prefix packages/web-app run test:e2e:ui + +# Full rename/autosave/commit/apply-ddl loop (selectors flagged as TODO): +E2E_FULL=1 npm --prefix packages/web-app run test:e2e +``` + +## Offline / air-gapped + +`OFFLINE=1 npm run test:e2e` short-circuits the clone so the rest of +the test tooling still compiles. You'll need a real parsed checkout for +the critical-path spec to pass — there's no local fallback by design. + +## Why this isn't in CI + +Running the full suite in CI would require: (1) a live network clone +of jaffle-shop, (2) a Python dbt-core install, (3) `dbt parse` to emit +`manifest.json`, (4) a Playwright browser install, (5) a real webServer +boot. That's multiple minutes of setup per PR plus flakiness on any +link in the chain. The backend contract it guards is already covered by +the api-server unit tests; the UI path it guards is caught faster by +dev-local runs while iterating. If we later want a CI-safe smoke, the +right move is to check in a tiny pre-parsed fixture instead of pulling +from the network. diff --git a/packages/web-app/e2e/critical-path.spec.js b/packages/web-app/e2e/critical-path.spec.js new file mode 100644 index 0000000..f0d749f --- /dev/null +++ b/packages/web-app/e2e/critical-path.spec.js @@ -0,0 +1,144 @@ +/* critical-path.spec.js — one big serial UI walk. + * + * Covers the "real user" journey on a cloned jaffle-shop checkout. + * Steps gated with `E2E_FULL=1` require UI selectors that haven't + * been pinned against the live app yet — enable those after a manual + * walkthrough pass. The smoke portion (import + explorer populate + + * Apply dialog wiring) runs unconditionally. + * + * A follow-up PR should drop the gate and wire concrete selectors + * for rename-cascade / autosave / auto-commit. + */ +import { test, expect } from "@playwright/test"; +import { execSync } from "node:child_process"; + +const FULL = process.env.E2E_FULL === "1"; + +test.describe.configure({ mode: "serial" }); + +test.describe("DataLex critical path on jaffle-shop", () => { + let projectDir; + + test.beforeAll(() => { + projectDir = process.env.JAFFLE_SHOP_DIR; + if (!projectDir) { + throw new Error( + "JAFFLE_SHOP_DIR not set — global-setup did not run. Use OFFLINE=1 to skip E2E." + ); + } + }); + + test("smoke: import jaffle-shop, explorer populates, Apply dialog wired", async ({ page }) => { + await page.addInitScript(() => { + localStorage.setItem( + "datalex.onboarding.seen", + JSON.stringify({ seen: true, seenAt: Date.now() }) + ); + }); + await page.goto("/"); + await expect(page.locator("body")).toBeVisible(); + + await test.step("open Import dbt dialog via Cmd+K", async () => { + await page.keyboard.press(process.platform === "darwin" ? "Meta+K" : "Control+K"); + await page.getByPlaceholder(/search|command/i).first().fill("Import dbt"); + await page.getByText(/Import dbt repo/i).first().click(); + await expect(page.getByRole("heading", { name: /Import dbt repo/i })).toBeVisible(); + }); + + await test.step("import from local jaffle-shop clone", async () => { + await page.getByRole("button", { name: /Local folder/i }).click(); + await page.getByLabel(/Local folder/i).fill(projectDir); + await page.getByRole("button", { name: /^Import$/i }).click(); + await expect(page.getByRole("heading", { name: /Import complete/i })).toBeVisible({ + timeout: 60_000, + }); + }); + + await test.step("open project from Results panel", async () => { + await page.getByRole("button", { name: /open project/i }).first().click(); + await expect(page.getByRole("heading", { name: /Import complete/i })).toBeHidden(); + }); + + await test.step("explorer lists jaffle-shop models", async () => { + await expect( + page.getByText(/stg_customers|customers\.ya?ml|orders\.ya?ml/i).first() + ).toBeVisible({ timeout: 20_000 }); + }); + + await test.step("Apply-to-warehouse dialog is wired", async () => { + await page.keyboard.press(process.platform === "darwin" ? "Meta+K" : "Control+K"); + await page.getByPlaceholder(/search|command/i).first().fill("Apply to warehouse"); + await page.getByText(/Apply to warehouse/i).first().click(); + await expect(page.getByRole("heading", { name: /Apply to warehouse/i })).toBeVisible(); + await expect(page.getByRole("button", { name: /Generate DDL/i })).toBeVisible(); + }); + }); + + test("full loop: rename cascade + autosave + auto-commit + DDL dry run", async ({ page }) => { + test.skip(!FULL, "Set E2E_FULL=1 to run selector-heavy steps (rename, autosave, auto-commit, DDL dry run)."); + + await page.addInitScript(() => { + localStorage.setItem( + "datalex.onboarding.seen", + JSON.stringify({ seen: true, seenAt: Date.now() }) + ); + }); + await page.goto("/"); + + // Prelude: reach a project-opened state reusing the smoke path. + await page.keyboard.press(process.platform === "darwin" ? "Meta+K" : "Control+K"); + await page.getByPlaceholder(/search|command/i).first().fill("Import dbt"); + await page.getByText(/Import dbt repo/i).first().click(); + await page.getByRole("button", { name: /Local folder/i }).click(); + await page.getByLabel(/Local folder/i).fill(projectDir); + await page.getByRole("button", { name: /^Import$/i }).click(); + await expect(page.getByRole("heading", { name: /Import complete/i })).toBeVisible({ timeout: 60_000 }); + await page.getByRole("button", { name: /open project/i }).first().click(); + + await test.step("rename entity cascades across siblings", async () => { + // TODO: pin selectors after a manual UI walkthrough. + // - Click a model file in the Explorer + // - Open inspector → Entity tab + // - Click "Rename entity…" button + // - Enter new name in the rename dialog → Rename + // - Assert toast with "Renamed … in N files" appears + throw new Error("selectors for rename-entity flow not yet pinned"); + }); + + await test.step("field edit autosaves within 1.5s", async () => { + // TODO: selectors for inspector column-type select. + // - Change a column type via Inspector → Columns + // - Wait 1500ms + // - await page.reload() + // - Assert the new type is still present + throw new Error("selectors for field-edit flow not yet pinned"); + }); + + await test.step("auto-commit coalesces bursty edits into one commit", async () => { + const before = countCommits(projectDir); + // TODO: CommitDialog → toggle auto-commit; perform 3 rapid edits; + // wait ~3s for the 2s debounce to fire. + const after = countCommits(projectDir); + expect(after - before).toBe(1); + }); + + await test.step("Apply to warehouse: generate DDL + dry run", async () => { + await page.keyboard.press(process.platform === "darwin" ? "Meta+K" : "Control+K"); + await page.getByPlaceholder(/search|command/i).first().fill("Apply to warehouse"); + await page.getByText(/Apply to warehouse/i).first().click(); + // Dialect dropdown → duckdb (stable for CI; no creds needed). + await page.locator("select").first().selectOption({ label: "DuckDB" }).catch(() => {}); + await page.getByRole("button", { name: /Generate DDL/i }).click(); + await expect(page.getByText(/Generated DDL/i)).toBeVisible({ timeout: 30_000 }); + }); + }); +}); + +function countCommits(dir) { + try { + const out = execSync("git rev-list --count HEAD", { cwd: dir, encoding: "utf8" }); + return parseInt(out.trim(), 10) || 0; + } catch { + return 0; + } +} diff --git a/packages/web-app/e2e/global-setup.js b/packages/web-app/e2e/global-setup.js new file mode 100644 index 0000000..ecd66ce --- /dev/null +++ b/packages/web-app/e2e/global-setup.js @@ -0,0 +1,48 @@ +/* Playwright global setup — clone jaffle-shop once, reuse everywhere. + * + * Cloning on every test would hammer GitHub and burn CI minutes. We + * clone once into `test-results/jaffle-shop/` (outside the package so + * Vite doesn't watch it) and skip re-cloning if the checkout looks + * intact. Tests read the cached path via process.env.JAFFLE_SHOP_DIR. + * + * Fails loud when offline — the suite relies on the real upstream repo + * as the fixture. Set OFFLINE=1 to skip the whole E2E suite in that + * environment instead of trying to shim the data. + */ +import { execSync } from "node:child_process"; +import { existsSync, mkdirSync, rmSync } from "node:fs"; +import { join, resolve } from "node:path"; + +const JAFFLE_URL = "https://github.com/dbt-labs/jaffle-shop.git"; +const JAFFLE_REF = process.env.JAFFLE_SHOP_REF || "main"; + +export default async function globalSetup() { + const cacheRoot = resolve(process.cwd(), "test-results"); + const dir = join(cacheRoot, "jaffle-shop"); + + mkdirSync(cacheRoot, { recursive: true }); + + const alreadyCloned = + existsSync(join(dir, "dbt_project.yml")) && existsSync(join(dir, ".git")); + + if (!alreadyCloned) { + if (existsSync(dir)) rmSync(dir, { recursive: true, force: true }); + console.log(`[e2e] cloning ${JAFFLE_URL}@${JAFFLE_REF} → ${dir}`); + try { + execSync( + `git clone --depth 1 --branch ${JAFFLE_REF} ${JAFFLE_URL} "${dir}"`, + { stdio: "inherit" } + ); + } catch (err) { + throw new Error( + `Failed to clone jaffle-shop. Set OFFLINE=1 to skip E2E, or check network / GitHub access.\n${err.message}` + ); + } + } else { + console.log(`[e2e] reusing cached jaffle-shop at ${dir}`); + } + + process.env.JAFFLE_SHOP_DIR = dir; + process.env.JAFFLE_SHOP_URL = JAFFLE_URL; + process.env.JAFFLE_SHOP_REF = JAFFLE_REF; +} diff --git a/packages/web-app/e2e/import-api.spec.js b/packages/web-app/e2e/import-api.spec.js new file mode 100644 index 0000000..5f62cd0 --- /dev/null +++ b/packages/web-app/e2e/import-api.spec.js @@ -0,0 +1,46 @@ +/* import-api.spec.js — backend contract regression gate. + * + * Stable, DOM-free coverage of the /api/dbt/import surface against a + * real jaffle-shop clone. This catches api-server/core-engine breakage + * independent of selector drift in the UI specs. + */ +import { test, expect } from "@playwright/test"; + +const API = "http://localhost:3006"; + +test.describe("dbt import API against real jaffle-shop", () => { + test("POST /api/dbt/import (local folder, skip warehouse) returns tree + report", async ({ request }) => { + const projectDir = process.env.JAFFLE_SHOP_DIR; + test.skip(!projectDir, "global-setup did not expose JAFFLE_SHOP_DIR"); + + const res = await request.post(`${API}/api/dbt/import`, { + data: { + projectDir, + skipWarehouse: true, + editInPlace: false, + }, + timeout: 60_000, + }); + expect(res.status(), await res.text()).toBe(200); + const body = await res.json(); + + // Tree: at minimum the canonical jaffle-shop models show up. + expect(Array.isArray(body.tree)).toBe(true); + expect(body.tree.length).toBeGreaterThan(3); + const paths = body.tree.map((f) => f.path); + const joined = paths.join("\n"); + expect(joined).toMatch(/customers/i); + expect(joined).toMatch(/orders/i); + + // Report: the SyncReport shape Phase 2 depends on. + expect(body.report).toBeTruthy(); + expect(typeof body.report).toBe("object"); + }); + + test("POST /api/dbt/import rejects missing projectDir", async ({ request }) => { + const res = await request.post(`${API}/api/dbt/import`, { + data: { skipWarehouse: true }, + }); + expect(res.status()).toBeGreaterThanOrEqual(400); + }); +}); diff --git a/packages/web-app/package-lock.json b/packages/web-app/package-lock.json index 001a6a7..4da0231 100644 --- a/packages/web-app/package-lock.json +++ b/packages/web-app/package-lock.json @@ -1,12 +1,12 @@ { "name": "datalex-web-app", - "version": "1.0.4", + "version": "1.0.6", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "datalex-web-app", - "version": "1.0.4", + "version": "1.0.6", "dependencies": { "@codemirror/lang-yaml": "^6.1.2", "@codemirror/lint": "^6.9.3", @@ -30,6 +30,7 @@ "zustand": "^5.0.11" }, "devDependencies": { + "@playwright/test": "^1.48.0", "@vitejs/plugin-react": "^4.3.4", "vite": "^5.4.10" } @@ -887,6 +888,22 @@ "integrity": "sha512-l0h88YhZFyKdXIFNfSWpyjStDjGHwZ/U7iobcK1cQQD8sejsONdQtTVU+1wVN1PBw40PiiHB1vA5S7VTfQiP9g==", "license": "MIT" }, + "node_modules/@playwright/test": { + "version": "1.59.1", + "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.59.1.tgz", + "integrity": "sha512-PG6q63nQg5c9rIi4/Z5lR5IVF7yU5MqmKaPOe0HSc0O2cX1fPi96sUQu5j7eo4gKCkB2AnNGoWt7y4/Xx3Kcqg==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright": "1.59.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/@rolldown/pluginutils": { "version": "1.0.0-beta.27", "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-beta.27.tgz", @@ -2500,6 +2517,53 @@ "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", "license": "ISC" }, + "node_modules/playwright": { + "version": "1.59.1", + "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.59.1.tgz", + "integrity": "sha512-C8oWjPR3F81yljW9o5OxcWzfh6avkVwDD2VYdwIGqTkl+OGFISgypqzfu7dOe4QNLL2aqcWBmI3PMtLIK233lw==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright-core": "1.59.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "fsevents": "2.3.2" + } + }, + "node_modules/playwright-core": { + "version": "1.59.1", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.59.1.tgz", + "integrity": "sha512-HBV/RJg81z5BiiZ9yPzIiClYV/QMsDCKUyogwH9p3MCP6IYjUFu/MActgYAvK0oWyV9NlwM3GLBjADyWgydVyg==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "playwright-core": "cli.js" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/playwright/node_modules/fsevents": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", + "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, "node_modules/postcss": { "version": "8.5.6", "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.6.tgz", diff --git a/packages/web-app/package.json b/packages/web-app/package.json index 983f2b2..6d672bb 100644 --- a/packages/web-app/package.json +++ b/packages/web-app/package.json @@ -7,7 +7,9 @@ "dev": "vite", "build": "vite build", "preview": "vite preview", - "test": "node --test tests/*.test.js" + "test": "node --test tests/*.test.js", + "test:e2e": "playwright test", + "test:e2e:ui": "playwright test --ui" }, "dependencies": { "@codemirror/lang-yaml": "^6.1.2", @@ -32,6 +34,7 @@ "zustand": "^5.0.11" }, "devDependencies": { + "@playwright/test": "^1.48.0", "@vitejs/plugin-react": "^4.3.4", "vite": "^5.4.10" } diff --git a/packages/web-app/playwright.config.js b/packages/web-app/playwright.config.js new file mode 100644 index 0000000..cc9a8f7 --- /dev/null +++ b/packages/web-app/playwright.config.js @@ -0,0 +1,65 @@ +/* Playwright config for the DataLex web-app E2E suite. + * + * Tests drive the real user journey — clone jaffle-shop from GitHub, + * import it via the dbt dialog, rename an entity, edit a field, verify + * autosave + auto-commit + apply-to-warehouse dry run. A global-setup + * clones jaffle-shop once into `test-results/jaffle-shop` and reuses + * that checkout across tests so we don't pound GitHub on every run. + * + * Both the api-server (port 3006) and the Vite dev server (port 5173) + * are started by Playwright's `webServer` block. The Vite proxy at + * `/api -> localhost:3006` matches what developers use locally. + */ +import { defineConfig, devices } from "@playwright/test"; + +const CI = !!process.env.CI; +// Opt-out for fully offline environments — set OFFLINE=1 to skip. +const SKIP = process.env.OFFLINE === "1"; + +export default defineConfig({ + testDir: "./e2e", + timeout: 120_000, + expect: { timeout: 15_000 }, + // E2E setup + state (repo cache) is shared; run specs serially. + fullyParallel: false, + workers: 1, + forbidOnly: CI, + retries: CI ? 1 : 0, + reporter: CI ? [["list"], ["html", { open: "never" }]] : [["list"]], + globalSetup: SKIP ? undefined : "./e2e/global-setup.js", + use: { + baseURL: "http://localhost:5173", + trace: "retain-on-failure", + screenshot: "only-on-failure", + video: CI ? "retain-on-failure" : "off", + actionTimeout: 10_000, + navigationTimeout: 30_000, + }, + projects: [ + { + name: "chromium", + use: { ...devices["Desktop Chrome"] }, + }, + ], + webServer: [ + { + command: "node index.js", + cwd: "../api-server", + port: 3006, + reuseExistingServer: !CI, + timeout: 60_000, + env: { + // Enable the "Apply to warehouse" endpoint so the dry-run test + // doesn't hit the 403 gate. Safe for tests — we only run the + // DuckDB dialect against a throwaway project. + DM_ENABLE_DIRECT_APPLY: "1", + }, + }, + { + command: "npm run dev", + port: 5173, + reuseExistingServer: !CI, + timeout: 60_000, + }, + ], +}); diff --git a/packages/web-app/src/components/dialogs/ImportDbtRepoDialog.jsx b/packages/web-app/src/components/dialogs/ImportDbtRepoDialog.jsx index f16bb5c..3372474 100644 --- a/packages/web-app/src/components/dialogs/ImportDbtRepoDialog.jsx +++ b/packages/web-app/src/components/dialogs/ImportDbtRepoDialog.jsx @@ -1,77 +1,37 @@ /* Import dbt Repo — entry point for the "I just want to try this with my dbt - * project" flow. Three input modes share one submit path: + * project" flow. Two input modes share one submit path: * 1. Local folder (absolute path on the server) * 2. Public git URL + optional ref - * 3. "Load jaffle-shop demo" — one-click load of a known-good dbt project * * On submit we call `POST /api/dbt/import` (wraps `dm dbt import`) which * returns `{tree: [{path, content}], report, project?}`. For local-folder * imports with "Edit in place" checked, the server also registers the folder * as a DataLex project and returns it; the web-app then binds the tree to * that project so Save All writes edits back into the original dbt repo - * at each file's source path. Without "Edit in place" (or for git/demo - * imports), the tree lives in memory only. + * at each file's source path. Without "Edit in place" (or for git imports), + * the tree lives in memory only. * - * The jaffle-shop demo tries a checked-in local fixture first (via Vite's - * `import.meta.glob`) and falls back to the public git URL so `dm serve` - * works offline when the fixture is present and degrades gracefully when - * it isn't. + * To try DataLex with a canonical dbt project, paste the public jaffle-shop + * repo URL into the Git URL tab: https://github.com/dbt-labs/jaffle-shop */ import React, { useState } from "react"; -import { GitBranch, FolderOpen, Sparkles, AlertCircle, Loader2 } from "lucide-react"; +import { GitBranch, FolderOpen, AlertCircle, Loader2 } from "lucide-react"; import Modal from "./Modal"; +import ImportResultsPanel from "./ImportResultsPanel"; import useUiStore from "../../stores/uiStore"; import useWorkspaceStore from "../../stores/workspaceStore"; import { importDbtProject } from "../../lib/api"; -// Vite-native glob-import of the checked-in jaffle-shop fixture. -// `{as: "raw"}` returns the file content as a plain string (YAML text), and -// the `eager: true` default we set below loads all files at build time so -// "Load demo" is instant and offline-ready. The bundle cost is ~few KB -// because the fixture is small — much cheaper than a round-trip to -// GitHub every time a user clicks the button. -// -// When the fixture directory doesn't exist, the glob resolves to an empty -// object and the demo falls back to the network path automatically. -const JAFFLE_FIXTURE = (() => { - try { - // eslint-disable-next-line no-undef - return import.meta.glob("../../fixtures/jaffle-shop/**/*.{yaml,yml}", { - query: "?raw", - import: "default", - eager: true, - }); - } catch (_err) { - return {}; - } -})(); - -const JAFFLE_GIT_URL = "https://github.com/dbt-labs/jaffle-shop.git"; -const JAFFLE_GIT_REF = "main"; - -function loadJaffleFixture() { - // With `eager: true`, values are strings — no async loader to await. - const entries = Object.entries(JAFFLE_FIXTURE); - if (entries.length === 0) return null; - const tree = []; - for (const [fullPath, content] of entries) { - const rel = fullPath.replace(/^.*\/fixtures\/jaffle-shop\//, ""); - tree.push({ path: rel, content: String(content || "") }); - } - return tree.length > 0 ? tree : null; -} - const TABS = [ - { id: "demo", label: "Demo", icon: Sparkles }, - { id: "folder", label: "Local folder", icon: FolderOpen }, { id: "git", label: "Git URL", icon: GitBranch }, + { id: "folder", label: "Local folder", icon: FolderOpen }, ]; export default function ImportDbtRepoDialog() { const { closeModal, addToast } = useUiStore(); const { loadDbtImportTree, loadDbtImportTreeAsProject } = useWorkspaceStore(); - const [tab, setTab] = useState("demo"); + const [tab, setTab] = useState("git"); // Folder mode const [folder, setFolder] = useState(""); @@ -90,36 +50,23 @@ export default function ImportDbtRepoDialog() { const [progress, setProgress] = useState(""); // human-readable current step const [error, setError] = useState(""); + // Import Results panel state — after a successful import we stash the + // SyncReport + tree and keep the dialog open so the user can inspect + // gaps (unknown types, unresolved rels, manifest-only banner) before + // jumping into the canvas. `openProject` is the deferred action we + // invoke when the user clicks "Open project" in the panel. + const [results, setResults] = useState(null); // { report, tree, sourceLabel, openProject } + const canSubmit = !submitting && - ((tab === "demo") || - (tab === "folder" && folder.trim()) || + ((tab === "folder" && folder.trim()) || (tab === "git" && gitUrl.trim())); - const ingestTree = async (tree, label) => { - await loadDbtImportTree(tree, { sourceLabel: label }); - addToast({ - type: "success", - message: `Loaded ${tree.length} file${tree.length === 1 ? "" : "s"} from ${label}.`, - }); - closeModal(); - }; - - const handleDemo = async () => { - setProgress("Loading bundled jaffle-shop fixture…"); - const local = loadJaffleFixture(); - if (local) { - await ingestTree(local, "jaffle-shop demo"); - return; - } - // Fall back to network import via the api-server. - setProgress("No bundled fixture — cloning from GitHub…"); - const res = await importDbtProject({ - gitUrl: JAFFLE_GIT_URL, - gitRef: JAFFLE_GIT_REF, - skipWarehouse: true, - }); - await ingestTree(res.tree || [], "jaffle-shop (github)"); + // Show the Import Results panel instead of closing. The actual load of + // the tree into the workspace is deferred to the panel's "Open project" + // button via `openProject`. + const showResults = ({ tree, report, sourceLabel, openProject }) => { + setResults({ tree: tree || [], report: report || null, sourceLabel, openProject }); }; const handleFolder = async () => { @@ -131,39 +78,53 @@ export default function ImportDbtRepoDialog() { editInPlace: !!editInPlace, }); const tree = res.tree || []; - // When the api-server registered a project for us, bind the tree to it so - // Save All writes back into the dbt repo. Otherwise fall back to the - // in-memory loader (explore-only). - if (editInPlace && res.project && res.project.id) { - await loadDbtImportTreeAsProject(tree, res.project); - const n = tree.length; - addToast({ - type: "success", - message: `Opened ${dir} in place — ${n} file${n === 1 ? "" : "s"}. Save All writes back into this folder.`, - }); - // Collision warning: shared schema.yml files will clobber sibling - // models on save until the Phase-2 merge path lands. - const collisions = useWorkspaceStore.getState().dbtImportCollisions || []; - if (collisions.length) { + + const openProject = async () => { + if (editInPlace && res.project && res.project.id) { + await loadDbtImportTreeAsProject(tree, res.project); + addToast({ + type: "success", + message: `Opened ${dir} in place — ${tree.length} file${tree.length === 1 ? "" : "s"}. Save All writes back into this folder.`, + }); + const collisions = useWorkspaceStore.getState().dbtImportCollisions || []; + if (collisions.length) { + addToast({ + type: "warning", + message: `${collisions.length} shared schema file${collisions.length === 1 ? "" : "s"} detected; saves may overwrite sibling models. See Save All preview.`, + }); + } + } else { + await loadDbtImportTree(tree, { sourceLabel: dir }); addToast({ - type: "warning", - message: `${collisions.length} shared schema file${collisions.length === 1 ? "" : "s"} detected; saves may overwrite sibling models. See Save All preview.`, + type: "success", + message: `Loaded ${tree.length} file${tree.length === 1 ? "" : "s"} from ${dir}.`, }); } - closeModal(); - return; - } - await ingestTree(tree, dir); + }; + + showResults({ tree, report: res.report || null, sourceLabel: dir, openProject }); }; const handleGit = async () => { - setProgress(`Cloning ${gitUrl.trim()}…`); + const label = gitUrl.trim(); + setProgress(`Cloning ${label}…`); const res = await importDbtProject({ - gitUrl: gitUrl.trim(), + gitUrl: label, gitRef: gitRef.trim() || "main", skipWarehouse, }); - await ingestTree(res.tree || [], gitUrl.trim()); + showResults({ + tree: res.tree || [], + report: res.report || null, + sourceLabel: label, + openProject: async () => { + await loadDbtImportTree(res.tree || [], { sourceLabel: label }); + addToast({ + type: "success", + message: `Loaded ${(res.tree || []).length} file${(res.tree || []).length === 1 ? "" : "s"} from ${label}.`, + }); + }, + }); }; const handleSubmit = async (e) => { @@ -173,8 +134,7 @@ export default function ImportDbtRepoDialog() { setError(""); setProgress(""); try { - if (tab === "demo") await handleDemo(); - else if (tab === "folder") await handleFolder(); + if (tab === "folder") await handleFolder(); else if (tab === "git") await handleGit(); } catch (err) { setError(err?.message || String(err) || "Import failed."); @@ -184,6 +144,38 @@ export default function ImportDbtRepoDialog() { } }; + // Once the import resolves we swap the form out for the Results panel. + if (results) { + const handleOpen = async () => { + try { + if (results.openProject) await results.openProject(); + } finally { + closeModal(); + } + }; + return ( + } + title="Import complete" + subtitle="Review the report below, then open the project." + size="lg" + onClose={closeModal} + footer={ + + } + > + + + ); + } + return ( } @@ -212,8 +204,6 @@ export default function ImportDbtRepoDialog() { Importing… - ) : tab === "demo" ? ( - "Load demo" ) : ( "Import" )} @@ -267,49 +257,6 @@ export default function ImportDbtRepoDialog() { })} - {tab === "demo" && ( -
-
-
- -
-
-
- dbt-labs / jaffle-shop -
-
- The canonical dbt demo project — staging models, marts, seeds, tests. - Great for exploring DataLex without wiring up your own repo. -
-
- {Object.keys(JAFFLE_FIXTURE).length > 0 - ? `${Object.keys(JAFFLE_FIXTURE).length} bundled files • offline` - : `via ${JAFFLE_GIT_URL.replace(/\.git$/, "")}`} -
-
-
-
- )} - {tab === "folder" && (
@@ -398,28 +346,26 @@ export default function ImportDbtRepoDialog() { )} - {(tab === "folder" || tab === "git") && ( -
- +
{progress && !error && (
{[ - "Import your dbt repo (folder / git URL / jaffle-shop demo)", + "Import your dbt repo (local folder or public git URL)", "Build your first diagram with auto-layout", "Wire relationships with inline endpoint validation", "Save All — merge-safe, git-diff-ready", diff --git a/packages/web-app/src/design/Shell.jsx b/packages/web-app/src/design/Shell.jsx index ba477e7..825f450 100644 --- a/packages/web-app/src/design/Shell.jsx +++ b/packages/web-app/src/design/Shell.jsx @@ -1050,7 +1050,6 @@ export default function Shell() { { id: "connect", section: "Actions", label: "Manage connections…", meta: "", icon: ⛁, run: () => openModal("connectionsManager") }, { id: "import", section: "Actions", label: "Import schema…", meta: "", icon: ⇩, run: () => openModal("importDialog") }, { id: "import-dbt", section: "Actions", label: "Import dbt repo…", meta: "", icon: ⤓, run: () => openModal("importDbtRepo") }, - { id: "demo-jaffle",section: "Actions", label: "Load jaffle-shop demo", meta: "", icon: ✨, run: () => openModal("importDbtRepo") }, { id: "apply-ddl", section: "Actions", label: "Apply to warehouse…", meta: "", icon: ☁, run: () => openModal("applyDdl") }, // v0.5.0 — stakeholder-share + snapshot flows. Share opens the // HTML bundle dialog prefilled from the currently-adapted schema; diff --git a/packages/web-app/src/fixtures/jaffle-shop/README.md b/packages/web-app/src/fixtures/jaffle-shop/README.md deleted file mode 100644 index 6fe927b..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# jaffle-shop fixture - -Bundled "Load demo" dataset for the **Import dbt repo** dialog. - -These YAML files were produced by running `dm dbt import` against -[dbt-labs/jaffle-shop](https://github.com/dbt-labs/jaffle-shop) and are -checked in so the demo loads instantly without a network round-trip. - -## Regenerating - -```bash -cd /path/to/jaffle-shop -dbt deps -dbt parse # writes target/manifest.json - -dm dbt import \ - --project-dir /path/to/jaffle-shop \ - --out packages/web-app/src/fixtures/jaffle-shop \ - --skip-warehouse -``` - -## Shape - -Folder layout mirrors the dbt repo (`models/staging/jaffle_shop/`, -`models/marts/`, `models/metrics/`). Each file is DataLex-shaped -(`kind: model`, `columns: [...]`, `meta.datalex.dbt.unique_id`) so the -Explorer / canvas / right panel all render as they would for a real -project. - -`--skip-warehouse` intentionally leaves columns without `data_type`, which -is the default for dbt users who haven't configured a warehouse — this -also exercises the dbt lint rules (missing type / missing tests) so the -demo showcases validation alongside the tree render. diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/marts/dim_customers.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/marts/dim_customers.yaml deleted file mode 100644 index d11c8c2..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/marts/dim_customers.yaml +++ /dev/null @@ -1,35 +0,0 @@ -kind: model -name: dim_customers -materialization: table -database: jaffle_shop -schema: main -description: Customer overview data mart, offering key details for each unique customer. - One row per customer. -depends_on: -- ref: stg_customers -- ref: fct_orders -- ref: order_items -columns: -- name: customer_id - description: The unique key of the orders mart. -- name: customer_name - description: Customers' full name. -- name: count_lifetime_orders - description: Total number of orders a customer has ever placed. -- name: first_ordered_at - description: The timestamp when a customer placed their first order. -- name: last_ordered_at - description: The timestamp of a customer's most recent order. -- name: lifetime_spend_pretax - description: The sum of all the pre-tax subtotals of every order a customer has - placed. -- name: lifetime_spend - description: The sum of all the order totals (including tax) that a customer has - ever placed. -- name: customer_type - description: Options are 'new' or 'returning', indicating if a customer has ordered - more than once or has only placed their first order to date. -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.dim_customers diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/marts/fct_orders.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/marts/fct_orders.yaml deleted file mode 100644 index 1b316b4..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/marts/fct_orders.yaml +++ /dev/null @@ -1,50 +0,0 @@ -kind: model -name: fct_orders -materialization: table -database: jaffle_shop -schema: main -description: Order overview data mart, offering key details for each order inlcluding - if it's a customer's first order and a food vs. drink item breakdown. One row per - order. -depends_on: -- ref: stg_orders -- ref: order_items -columns: -- name: order_id - description: The unique key of the orders mart. -- name: customer_id - description: The foreign key relating to the customer who placed the order. -- name: location_id - description: The foreign key relating to the location the order was placed at. -- name: order_total - description: The total amount of the order in USD including tax. -- name: ordered_at - description: The timestamp the order was placed at. -- name: count_food_items - description: The number of individual food items ordered. -- name: count_drink_items - description: The number of individual drink items ordered. -- name: count_items - description: The total number of both food and drink items ordered. -- name: subtotal_food_items - description: The sum of all the food item prices without tax. -- name: subtotal_drink_items - description: The sum of all the drink item prices without tax. -- name: subtotal - description: The sum total of both food and drink item prices without tax. -- name: order_cost - description: The sum of supply expenses to fulfill the order. -- name: location_name - description: The full location name of where this order was placed. Denormalized - from `stg_locations`. -- name: is_first_order - description: A boolean indicating if this order is from a new customer placing their - first order. -- name: is_food_order - description: A boolean indicating if this order included any food items. -- name: is_drink_order - description: A boolean indicating if this order included any drink items. -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.fct_orders diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/marts/order_items.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/marts/order_items.yaml deleted file mode 100644 index 8001c63..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/marts/order_items.yaml +++ /dev/null @@ -1,14 +0,0 @@ -kind: model -name: order_items -materialization: table -database: jaffle_shop -schema: main -depends_on: -- ref: stg_order_items -- ref: stg_orders -- ref: stg_products -- ref: stg_supplies -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.order_items diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/metrics/metricflow_time_spine.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/metrics/metricflow_time_spine.yaml deleted file mode 100644 index 8cb8219..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/metrics/metricflow_time_spine.yaml +++ /dev/null @@ -1,9 +0,0 @@ -kind: model -name: metricflow_time_spine -materialization: view -database: jaffle_shop -schema: main -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.metricflow_time_spine diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/jaffle_shop.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/jaffle_shop.yaml deleted file mode 100644 index 51ac2a4..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/jaffle_shop.yaml +++ /dev/null @@ -1,79 +0,0 @@ -kind: source -name: jaffle_shop -database: jaffle_shop -schema: new_jaffle_shop -tables: -- name: customers - freshness: - warn_after: - count: null - period: null - error_after: - count: null - period: null - filter: null - meta: - datalex: - dbt: - unique_id: source.jaffle_shop.jaffle_shop.customers -- name: orders - freshness: - warn_after: - count: null - period: null - error_after: - count: null - period: null - filter: null - meta: - datalex: - dbt: - unique_id: source.jaffle_shop.jaffle_shop.orders -- name: items - freshness: - warn_after: - count: null - period: null - error_after: - count: null - period: null - filter: null - meta: - datalex: - dbt: - unique_id: source.jaffle_shop.jaffle_shop.items -- name: products - freshness: - warn_after: - count: null - period: null - error_after: - count: null - period: null - filter: null - meta: - datalex: - dbt: - unique_id: source.jaffle_shop.jaffle_shop.products -- name: supplies - freshness: - warn_after: - count: null - period: null - error_after: - count: null - period: null - filter: null - meta: - datalex: - dbt: - unique_id: source.jaffle_shop.jaffle_shop.supplies -meta: - datalex: - dbt: - unique_ids: - - source.jaffle_shop.jaffle_shop.customers - - source.jaffle_shop.jaffle_shop.items - - source.jaffle_shop.jaffle_shop.orders - - source.jaffle_shop.jaffle_shop.products - - source.jaffle_shop.jaffle_shop.supplies diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_customers.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_customers.yaml deleted file mode 100644 index 2c457ed..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_customers.yaml +++ /dev/null @@ -1,18 +0,0 @@ -kind: model -name: stg_customers -materialization: view -database: jaffle_shop -schema: main -description: Customer data with basic cleaning and transformation applied, one row - per customer. -depends_on: -- source: - source: jaffle_shop - name: customers -columns: -- name: customer_id - description: The unique key for each customer. -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.stg_customers diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_order_items.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_order_items.yaml deleted file mode 100644 index 870d517..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_order_items.yaml +++ /dev/null @@ -1,18 +0,0 @@ -kind: model -name: stg_order_items -materialization: view -database: jaffle_shop -schema: main -description: Individual food and drink items that make up our orders, one row per - item. -depends_on: -- source: - source: jaffle_shop - name: items -columns: -- name: order_item_id - description: The unique key for each order item. -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.stg_order_items diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_orders.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_orders.yaml deleted file mode 100644 index 09f6bfb..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_orders.yaml +++ /dev/null @@ -1,18 +0,0 @@ -kind: model -name: stg_orders -materialization: table -database: jaffle_shop -schema: main -description: Order data with basic cleaning and transformation applied, one row per - order. -depends_on: -- source: - source: jaffle_shop - name: orders -columns: -- name: order_id - description: The unique key for each order. -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.stg_orders diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_products.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_products.yaml deleted file mode 100644 index 869ff2b..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_products.yaml +++ /dev/null @@ -1,18 +0,0 @@ -kind: model -name: stg_products -materialization: view -database: jaffle_shop -schema: main -description: Product (food and drink items that can be ordered) data with basic cleaning - and transformation applied, one row per product. -depends_on: -- source: - source: jaffle_shop - name: products -columns: -- name: product_id - description: The unique key for each product. -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.stg_products diff --git a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_supplies.yaml b/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_supplies.yaml deleted file mode 100644 index e6d6fa0..0000000 --- a/packages/web-app/src/fixtures/jaffle-shop/models/staging/jaffle_shop/stg_supplies.yaml +++ /dev/null @@ -1,23 +0,0 @@ -kind: model -name: stg_supplies -materialization: view -database: jaffle_shop -schema: main -description: 'List of our supply expenses data with basic cleaning and transformation - applied. - - One row per supply cost, not per supply. As supply costs fluctuate they receive - a new row with a new UUID. Thus there can be multiple rows per supply_id. - - ' -depends_on: -- source: - source: jaffle_shop - name: supplies -columns: -- name: supply_uuid - description: The unique key of our supplies per cost. -meta: - datalex: - dbt: - unique_id: model.jaffle_shop.stg_supplies diff --git a/packages/web-app/src/lib/onboardingTour.js b/packages/web-app/src/lib/onboardingTour.js index 9ef69c6..3d27886 100644 --- a/packages/web-app/src/lib/onboardingTour.js +++ b/packages/web-app/src/lib/onboardingTour.js @@ -35,7 +35,7 @@ const TOUR_STEPS = [ popover: { title: "1 · Import your dbt repo", description: - "Start here. Paste a git URL, pick a local folder, or load the bundled jaffle-shop demo. Every model in manifest.json becomes a DataLex YAML entity — the Explorer on the left fills in immediately.", + "Start here. Paste a public git URL (e.g. https://github.com/dbt-labs/jaffle-shop) or pick a local folder. Every model in manifest.json becomes a DataLex YAML entity — the Explorer on the left fills in immediately.", side: "bottom", align: "start", },