From ed97f4f2747e0420319d1e64449a89000ab15074 Mon Sep 17 00:00:00 2001 From: KKranthi6881 Date: Fri, 17 Apr 2026 16:07:46 -0500 Subject: [PATCH] docs: consolidate on Nextra site, drop legacy docs/ folder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User-facing documentation now lives in a single place: the Nextra site under apps/docs/ (published to docs.duckcode.ai). The legacy docs/ folder had 27 half-migrated files with stale content, broken cross-links, and references to deleted commands — confusing for new users. - Rewrite root README from 487 lines → ~55 lines. Single install path (npx create-dql-app), then point at docs.duckcode.ai. Drop the 25 broken deep-links into docs/. - Port 3 pages that were missing from Nextra: * guides/faq.mdx * reference/project-layout.mdx * reference/compatibility.mdx - Update reference/file-formats.mdx: cdql.yaml → dql.config.json (matches what v1.0.1 actually ships), clarify .dqlnb is the current notebook format, not legacy. - Delete 27 legacy docs/ files: the 01-/02-/03-/04-/05- numbered directories, migration-guides/, examples/, dql-language-spec, faq, compatibility, project-config, project-structure, v1-support-scope, publishing, repo-testing, data-sources. Content is covered in Nextra or was itself stale. - Keep docs/oss-readiness-checklist.md (internal maintainer checklist). - Move examples/jaffle-shop-lineage-demo.dqlnb to examples/gallery/ so all example assets live in one place. - Rewrite docs/README.md as a thin pointer to the Nextra site. - Nextra builds clean: 28 pages, link-check passes with 0 broken links. Co-Authored-By: Claude Opus 4.7 --- README.md | 485 +--------------- apps/docs/pages/guides/_meta.json | 3 +- apps/docs/pages/guides/faq.mdx | 69 +++ apps/docs/pages/reference/_meta.json | 4 +- apps/docs/pages/reference/compatibility.mdx | 51 ++ apps/docs/pages/reference/file-formats.mdx | 64 +- apps/docs/pages/reference/project-layout.mdx | 62 ++ docs/01-start-here/README.md | 28 - docs/01-start-here/dbt-jaffle-shop.md | 113 ---- .../install-and-first-project.md | 62 -- docs/01-start-here/run-from-source.md | 93 --- docs/02-core-workflows/README.md | 20 - .../block-authoring-workflow.md | 34 -- docs/02-core-workflows/lineage-workflow.md | 96 --- docs/02-core-workflows/notebook-workflow.md | 31 - .../semantic-layer-workflow.md | 30 - docs/03-guides/README.md | 13 - docs/04-reference/README.md | 11 - docs/05-maintainers/README.md | 17 - docs/README.md | 31 +- docs/compatibility.md | 183 ------ docs/data-sources.md | 546 ------------------ docs/dql-language-spec.md | 270 --------- docs/examples/README.md | 15 - docs/faq.md | 176 ------ docs/migration-guides/README.md | 29 - docs/migration-guides/dbt.md | 88 --- docs/migration-guides/raw-sql.md | 136 ----- docs/migration-guides/saved-bi-query.md | 116 ---- docs/project-config.md | 127 ---- docs/project-structure.md | 61 -- docs/publishing.md | 197 ------- docs/repo-testing.md | 115 ---- docs/v1-support-scope.md | 20 - .../gallery}/jaffle-shop-lineage-demo.dqlnb | 0 35 files changed, 262 insertions(+), 3134 deletions(-) create mode 100644 apps/docs/pages/guides/faq.mdx create mode 100644 apps/docs/pages/reference/compatibility.mdx create mode 100644 apps/docs/pages/reference/project-layout.mdx delete mode 100644 docs/01-start-here/README.md delete mode 100644 docs/01-start-here/dbt-jaffle-shop.md delete mode 100644 docs/01-start-here/install-and-first-project.md delete mode 100644 docs/01-start-here/run-from-source.md delete mode 100644 docs/02-core-workflows/README.md delete mode 100644 docs/02-core-workflows/block-authoring-workflow.md delete mode 100644 docs/02-core-workflows/lineage-workflow.md delete mode 100644 docs/02-core-workflows/notebook-workflow.md delete mode 100644 docs/02-core-workflows/semantic-layer-workflow.md delete mode 100644 docs/03-guides/README.md delete mode 100644 docs/04-reference/README.md delete mode 100644 docs/05-maintainers/README.md delete mode 100644 docs/compatibility.md delete mode 100644 docs/data-sources.md delete mode 100644 docs/dql-language-spec.md delete mode 100644 docs/examples/README.md delete mode 100644 docs/faq.md delete mode 100644 docs/migration-guides/README.md delete mode 100644 docs/migration-guides/dbt.md delete mode 100644 docs/migration-guides/raw-sql.md delete mode 100644 docs/migration-guides/saved-bi-query.md delete mode 100644 docs/project-config.md delete mode 100644 docs/project-structure.md delete mode 100644 docs/publishing.md delete mode 100644 docs/repo-testing.md delete mode 100644 docs/v1-support-scope.md rename {docs/examples => examples/gallery}/jaffle-shop-lineage-demo.dqlnb (100%) diff --git a/README.md b/README.md index 3bfc6994..edb007f5 100644 --- a/README.md +++ b/README.md @@ -5,483 +5,52 @@ [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](./LICENSE) [![Node](https://img.shields.io/badge/node-18%20%7C%2020%20%7C%2022-green)](https://nodejs.org) -- Your team copy-pastes the same query into 12 dashboards. They drift. Nobody knows which is right. -- Someone changes the SQL. The chart breaks. There was no review. -- AI generated a perfect query last Tuesday. It's gone now. -- "Is this metric still correct?" — nobody can answer that. +**Analytics notebooks on your dbt models.** Git-native. Local-first. Open source. -**DQL fixes that.** +DQL sits between **dbt** (modeling) and your BI tool (reporting). Every analytics answer — SQL, chart, params, tests, owner — lives in a single `.dql` file tracked in git. No more query sprawl, no more broken charts, no more lost work. -**DQL is an open-source analytics language, CLI, and notebook UI** — each analytics answer lives in a single `.dql` file: SQL + visualization config + owner + tests + parameters, all Git-trackable. No more query sprawl. No more broken charts. No more lost work. - -→ **[Start with the docs hub](./docs/README.md)** - -→ **[Why DQL exists and what problem it solves](./docs/why-dql.md)** - ---- - -## Install - -Requires Node 18, 20, or 22 (active LTS). - -```bash -npm install -g @duckcodeailabs/dql-cli -dql --version -``` - -### Path A: DQL-Only (no dbt required) - -The fastest path — local project, local notebook, zero dependencies beyond Node. - -```bash -dql init my-dql-project -cd my-dql-project -dql doctor -dql notebook -``` - -Your browser opens with a notebook connected to DuckDB in-memory. Write SQL cells, query local CSV/Parquet files, and create governed DQL blocks — all tracked in Git. - -→ **[Quickstart — zero to running notebook in 2 minutes](./docs/quickstart.md)** - -→ **[Install and first project](./docs/01-start-here/install-and-first-project.md)** - -→ **[Full getting started guide with the Jaffle Shop dbt project](./docs/01-start-here/dbt-jaffle-shop.md)** - -### Path B: dbt + Jaffle Shop (5 minutes) - -Use the Jaffle Shop dbt project to see DQL's full power — semantic metrics, lineage, and Block Studio. - -> **Note:** You must clone the dbt repo yourself. `dql init` does not download it automatically. - -```bash -# 1. Clone and build the dbt project -git clone https://github.com/dbt-labs/Semantic-Layer-Online-Course.git jaffle-shop -cd jaffle-shop -pip install dbt-duckdb && dbt deps && dbt build --profiles-dir . - -# 2. Initialize DQL (auto-detects dbt + DuckDB + semantic definitions) -npm install -g @duckcodeailabs/dql-cli -dql init . -dql doctor -dql notebook -``` - -`dql init .` auto-detects the dbt project, finds `jaffle_shop.duckdb`, and imports semantic metrics/dimensions from the dbt YAML files — all in one step. - -→ **[Full getting started guide with Jaffle Shop](./docs/getting-started.md)** - -→ **[Quickstart — zero to running notebook in 5 minutes](./docs/01-start-here/dbt-jaffle-shop.md)** - -### Path C: Enterprise — Your dbt Repo + Your Database - -Already have a dbt project and a Snowflake/Postgres/BigQuery warehouse? DQL layers on top. - -```bash -cd your-dbt-project -dql init . -dql notebook -# → Use the Connection Panel in the notebook to configure your database -# → Import semantic metrics from the notebook UI or CLI -``` - -→ **[Enterprise getting started — connect your database, import metrics, build blocks](./docs/enterprise-getting-started.md)** - -### Library packages (for embedding in your app) - -```bash -npm install @duckcodeailabs/dql-core@0.8.7 @duckcodeailabs/dql-compiler@0.8.7 @duckcodeailabs/dql-notebook@0.8.7 -``` - ---- - -## The Notebook - -`dql notebook` opens a local browser UI — no cloud, no Python, no setup beyond the CLI. - -- **SQL cells** — write and run SQL against local files, powered by DuckDB -- **Param cells** — live widgets (text, number, date, select) wired to `{{variable}}` references in SQL -- **Markdown cells** — annotate your analysis -- **Auto-charting** — bar and line charts detected from query shape, table/chart toggle -- **Export** — standalone HTML dashboard or `.dql` workbook file - -```bash -dql notebook # open notebook for current project -dql notebook ./my-dql-project # specify project path -dql new notebook "Revenue Analysis" # scaffold a new .dqlnb file -``` - -→ **[Notebook guide — cells, param widgets, variable substitution, export](./docs/notebook.md)** - ---- - -## Block Studio - -Block Studio is the notebook's built-in IDE for creating governed DQL blocks. Open it from the sidebar or by clicking any `.dql` file. - -- **Database explorer** — browse your tables and columns, click to insert into SQL -- **Semantic panel** — browse imported metrics and dimensions, click to insert references -- **Live validation** — syntax and semantic errors as you type -- **Run** — execute SQL against your connected database, see results in a table or chart -- **Test** — run block assertions (e.g., `assert row_count > 0`) and see pass/fail -- **Save** — write the block to `blocks/` with governance metadata (owner, domain, tags) - -→ **[Authoring blocks — create, test, certify, and commit custom and semantic blocks](./docs/authoring-blocks.md)** - ---- - -## DQL Blocks - -A DQL block is the core reusable unit — one file that holds everything needed to produce a trusted analytics answer. - -```dql -block "Revenue by Segment" { - domain = "revenue" - owner = "data-team" - tags = ["revenue", "segment", "quarterly"] - - params { - period = "current_quarter" - } - - query = """ - SELECT segment_tier AS segment, SUM(amount) AS revenue - FROM fct_revenue - WHERE fiscal_period = ${period} - GROUP BY segment_tier - ORDER BY revenue DESC - """ - - visualization { - chart = "bar" - x = segment - y = revenue - } - - tests { - assert row_count > 0 - } -} -``` - -Use `ref("block_name")` to declare explicit dependencies between blocks: - -```dql -block "Executive Summary" { - domain = "executive" - query = """ - SELECT * FROM ref("revenue_by_segment") - WHERE revenue > 10000 - """ -} -``` - -Scaffold, validate, preview, and build blocks with the CLI: - -```bash -dql new block "Pipeline Health" -dql parse blocks/pipeline_health.dql -dql preview blocks/pipeline_health.dql --open -dql build blocks/pipeline_health.dql -``` - -→ **[Full language reference — block syntax, chart types, params, tests, workbooks](./docs/dql-language-spec.md)** - ---- - -## Lineage & Trust Chains - -DQL tracks how data flows from source tables through dbt models, DQL blocks, business domains, and notebooks — the full open-source trust chain from raw data to analytics consumption. - -```bash -dql compile # generate dql-manifest.json -dql compile --dbt-manifest path/to/manifest.json # import dbt lineage as upstream -dql lineage # full project lineage with data flow DAG -dql lineage --search revenue # search lineage nodes by name -dql lineage --focus revenue_by_customer_type # focused subgraph around a node -dql lineage --dashboard "Jaffle Shop Lineage Demo" # what feeds a notebook/dashboard? -dql lineage --dbt # dbt-only graph slice -dql lineage raw_orders # upstream/downstream for a block -dql lineage orders # smart lookup: resolves to table:orders -dql lineage --table orders # explicit table lookup -dql lineage --metric total_revenue # explicit metric lookup -dql lineage --domain finance # domain-scoped view -dql lineage --impact orders # what breaks if this table changes? -dql lineage --trust-chain raw_orders exec_dashboard # certification at every hop -dql lineage --format json # export for CI/integrations -``` - -In the notebook UI, users can: - -- search lineage from the sidebar and open a focused graph instead of scanning the full project -- open a block in Block Studio and use the `Lineage` tab for upstream provenance and downstream notebook usage -- inspect dbt sources/models, DQL blocks, domains, and notebooks in one graph - -Cross-domain flow detection is built in, so DQL also shows when data crosses team boundaries: - -``` - Cross-Domain Flows: - data -> finance (1 edge(s)) - finance -> executive (1 edge(s)) -``` - -For a concrete end-to-end demo, copy the ready-made notebook asset: +## Get started — under 2 minutes ```bash -mkdir -p notebooks -cp docs/examples/jaffle-shop-lineage-demo.dqlnb notebooks/ -``` - -That demo is designed to show this OSS path clearly: - -```text -dbt source -> dbt model -> DQL block -> notebook -``` - -→ **[Lineage guide — ref(), trust chains, impact analysis, cross-domain flows](./docs/lineage.md)** -→ **[Lineage workflow — focused UI flow, CLI commands, Block Studio lineage](./docs/02-core-workflows/lineage-workflow.md)** -→ **[Demo assets — copyable Jaffle Shop lineage notebook](./docs/examples/README.md)** - ---- - -## Semantic Layer - -DQL includes a built-in semantic layer — define metrics, dimensions, hierarchies, and cubes in YAML files, then reference them in your notebooks and blocks. - -``` -semantic-layer/ - metrics/revenue.yaml - dimensions/segment.yaml - hierarchies/revenue_time.yaml - cubes/revenue_cube.yaml -``` - -Four provider modes are supported: - -| Provider | Source | Use when | -|---|---|---| -| `dql` (default) | Local `semantic-layer/` YAML files | Standalone projects, getting started | -| `dbt` | dbt `models/**/*.yml` semantic models | You already use dbt for transformations | -| `cubejs` | Cube.js `model/` or `schema/` definitions | You already use Cube for your semantic layer | -| `snowflake` | Snowflake semantic views (live connection) | You use Snowflake as your semantic layer | - -Configure in `dql.config.json`: - -```json -{ - "semanticLayer": { - "provider": "dql" - } -} -``` - -The notebook sidebar shows your semantic layer live — browse metrics, dimensions, and hierarchies, click to insert into SQL cells. - -→ **[Semantic layer guide — setup, providers, YAML format, notebook integration](./docs/semantic-layer-guide.md)** - ---- - -## Data Sources - -DQL supports **14 database connectors** out of the box. The default runtime uses DuckDB to query local CSV and Parquet files directly — no cloud warehouse needed to get started. - -```json -{ - "connections": { - "default": { - "driver": "duckdb", - "filepath": ":memory:" - } - } -} +npx create-dql-app my-project +cd my-project +npm install +npm run notebook ``` -Connect to Postgres, Snowflake, BigQuery, or any of the 14 supported databases — either edit `dql.config.json` directly or use the **Connection Panel** in the notebook UI to configure and hot-swap connections at runtime. - -→ **[Data sources guide — all 14 connectors with config fields](./docs/data-sources.md)** - ---- - -## Getting Started with dbt - -DQL is the **answer layer for dbt**. The recommended getting-started path uses the [Jaffle Shop semantic-layer course repo](https://github.com/dbt-labs/Semantic-Layer-Online-Course) — a real dataset with models, metrics, and semantic definitions that DQL can read directly. - -`dql init` auto-detects dbt projects and DuckDB files, scaffolding a minimal DQL project on top of your existing data. - -→ **[Getting started — full walkthrough with Jaffle Shop](./docs/getting-started.md)** -→ **[Enterprise getting started — your own dbt repo + database](./docs/enterprise-getting-started.md)** - ---- - -## CLI Reference - -Every command has a clear job: - -| Command | What it does | -|---|---| -| `dql init` | Initialize DQL in a project (auto-detects dbt) | -| `dql notebook` | Open the browser notebook for interactive SQL exploration | -| `dql new block` | Create a new `.dql` block file | -| `dql new notebook` | Create a new `.dqlnb` notebook file | -| `dql parse` | Validate syntax and run semantic analysis | -| `dql preview` | Compile and serve a block locally with live data | -| `dql build` | Compile to a static HTML bundle | -| `dql serve` | Serve a built bundle locally | -| `dql certify` | Check governance rules (owner, description, tags, domain) | -| `dql compile` | Generate project manifest (`dql-manifest.json`) with lineage and dependencies | -| `dql lineage` | Show data lineage, trust chains, impact analysis, cross-domain flows | -| `dql fmt` | Format a `.dql` file in place | -| `dql doctor` | Diagnose project setup, config, and runtime readiness | -| `dql --version` | Show CLI version | - -→ **[Full CLI reference — all commands, flags, and exit codes](./docs/cli-reference.md)** +Your browser opens a notebook at `http://localhost:5173`. DuckDB is running in-memory. Drop a CSV into `data/`, query it, save a block, commit. ---- - -## Migration - -Coming from raw SQL, dbt, or a saved BI query? - -```bash -dql migrate raw-sql -dql migrate dbt -dql migrate looker -``` - -→ **[Migration guides — from raw SQL, dbt, and BI tools](./docs/migration-guides/README.md)** - ---- - -## VS Code Extension - -Search **DQL Language Support** in the Extensions panel, or: - -```bash -code --install-extension dql.dql-language-support -``` - -Provides syntax highlighting, snippets, format-on-save, and LSP support (completions, hover, diagnostics). - ---- - -## Use Cases - -Not sure where to start? Pick your goal: - -- **Explore a dbt project interactively** → `dql init` + `dql notebook` -- **Author a reusable block** → `dql new block` + `dql preview` -- **Build a shareable dashboard** → `dql build` + `dql serve` -- **Migrate from raw SQL or dbt** → `dql migrate` -- **Connect enterprise database** → `dql notebook` + Connection Panel -- **Embed the parser in Node.js** → `@duckcodeailabs/dql-core` - -→ **[Use cases — recommended paths by goal](./docs/use-cases.md)** - ---- +Already have a **dbt project**? Run `create-dql-app` next to it — the scaffolder auto-detects `dbt_project.yml` as a sibling and wires `dql.config.json` accordingly. Then `dql sync dbt` imports your manifest. ## Documentation -### Start Here +**📘 [docs.duckcode.ai](https://docs.duckcode.ai)** — the single source of truth. -| Guide | What it covers | -|---|---| -| [Docs Hub](./docs/README.md) | Ordered documentation index for new users and contributors | -| [Install and First Project](./docs/01-start-here/install-and-first-project.md) | Clean first-time-user setup path | -| [Run From Source](./docs/01-start-here/run-from-source.md) | Clone, install, build, test, and smoke-test locally | -| [dbt + Jaffle Shop Walkthrough](./docs/01-start-here/dbt-jaffle-shop.md) | Step-by-step dbt-backed semantic-layer setup | -| [Quickstart](./docs/quickstart.md) | Short dbt-based path from setup to notebook | -| [Getting Started](./docs/getting-started.md) | Detailed walkthrough with Jaffle Shop dbt project | -| [Enterprise Getting Started](./docs/enterprise-getting-started.md) | Your own dbt repo + Snowflake/Postgres/BigQuery | +Quick links: -### Core Workflows +- [Install](https://docs.duckcode.ai/get-started/install) · [Quickstart](https://docs.duckcode.ai/get-started/quickstart) · [Concepts](https://docs.duckcode.ai/get-started/concepts) +- [Jaffle Shop walkthrough](https://docs.duckcode.ai/guides/jaffle-shop) · [Import dbt](https://docs.duckcode.ai/guides/import-dbt) · [Author a block](https://docs.duckcode.ai/guides/authoring-blocks) +- [CLI reference](https://docs.duckcode.ai/reference/cli) · [Language reference](https://docs.duckcode.ai/reference/language) · [Connectors](https://docs.duckcode.ai/reference/connectors) +- [Architecture](https://docs.duckcode.ai/architecture/overview) · [Contributing](https://docs.duckcode.ai/contribute/repo-layout) -| Guide | What it covers | -|---|---| -| [Notebook Guide](./docs/notebook.md) | Cell types, param widgets, variable refs, export | -| [Authoring Blocks](./docs/authoring-blocks.md) | Create, test, certify, and commit .dql blocks (custom + semantic) | -| [Lineage & Trust Chains](./docs/lineage.md) | ref() system, cross-domain flows, impact analysis, trust chains | -| [Semantic Layer](./docs/semantic-layer-guide.md) | Metrics, dimensions, cubes, dbt/Cube.js providers | +## What's in the box -### Reference +- **Notebook** — SQL + DQL cells with live results, charts, and params +- **Block Studio** — governed, versioned analytics blocks with lint + certify +- **Semantic layer** — import dbt metrics/dimensions; author your own +- **Lineage DAG** — table · block · notebook granularity with impact analysis +- **Git-native format** — canonical `.dql` serialization, `dql diff`, in-app git panel +- **15 connectors** — Postgres, DuckDB, Snowflake, BigQuery, Redshift, MySQL, and more +- **VS Code extension** — syntax, snippets, LSP (`code --install-extension dql.dql-language-support`) -| Guide | What it covers | -|---|---| -| [CLI Reference](./docs/cli-reference.md) | Every command, flag, and exit code | -| [Data Sources](./docs/data-sources.md) | All 14 database drivers with config fields | -| [Language Spec](./docs/dql-language-spec.md) | Full .dql syntax: blocks, charts, params, workbooks | -| [Project Config](./docs/project-config.md) | dql.config.json — connections, ports, defaults | +## What this repo does **not** include -### Migrate & Examples - -| Guide | What it covers | -|---|---| -| [Migration Guides](./docs/migration-guides/README.md) | From raw SQL, dbt, Looker, Tableau | -| [Use Cases](./docs/use-cases.md) | Recommended paths by goal | -| [Examples](./docs/examples.md) | Suggested learning path and block examples | -| [Why DQL](./docs/why-dql.md) | The problem, before/after, personas, DQL vs alternatives | - -### Help & Compatibility - -| Guide | What it covers | -|---|---| -| [FAQ](./docs/faq.md) | Common questions about scope, notebook, and compatibility | -| [Compatibility](./docs/compatibility.md) | Runtime, connector, and workflow support matrix | -| [Roadmap](./ROADMAP.md) | Planned features and known limitations | -| [Security](./SECURITY.md) | Vulnerability reporting and credential handling | - ---- - -## Package Reference - -All packages share a unified version number (`0.8.7`). - -| Package | Description | -|---|---| -| `@duckcodeailabs/dql-cli` | Public CLI — `dql init`, `dql notebook`, `dql lineage`, `dql preview`, `dql parse`, ... | -| `@duckcodeailabs/dql-core` | Lexer, parser, AST, semantic analysis, semantic layer, lineage engine, formatter | -| `@duckcodeailabs/dql-compiler` | IR lowering, ref() resolution, HTML/React/runtime code generation | -| `@duckcodeailabs/dql-governance` | Certification rules, cost estimation | -| `@duckcodeailabs/dql-project` | Git-backed block registry and project primitives | -| `@duckcodeailabs/dql-notebook` | Notebook document model and execution helpers | -| `@duckcodeailabs/dql-lsp` | Language Server Protocol implementation | -| `@duckcodeailabs/dql-runtime` | Browser runtime: data fetching, hot-reload | -| `@duckcodeailabs/dql-charts` | React SVG chart components | -| `@duckcodeailabs/dql-connectors` | Database connector layer (14 drivers) | - ---- - -## What This Repo Does Not Include - -- Natural-language / agentic block generation -- MCP runtime -- Approvals, run history, or product orchestration - -Those remain part of the closed DuckCode product. - ---- +Natural-language / agentic block generation, MCP runtime, hosted workspaces, RBAC, scheduled runs, and alerting all live in the closed DuckCode Cloud product. ## Contributing -Contributions are welcome. See [CONTRIBUTING.md](./CONTRIBUTING.md) for setup, code style, and PR guidelines. - -Areas where help is most useful: -- New database connector drivers -- Additional semantic layer providers -- Language spec improvements and test coverage -- Documentation and examples - ---- - -## Community and Support - -- **Bugs and feature requests** — [open a GitHub issue](https://github.com/duckcode-ai/dql/issues) -- **Questions and discussion** — [GitHub Discussions](https://github.com/duckcode-ai/dql/discussions) -- **Roadmap** — see [ROADMAP.md](./ROADMAP.md) for planned work and known limitations - -This project follows standard GitHub community norms. Please be respectful in all interactions. - ---- +See [CONTRIBUTING.md](./CONTRIBUTING.md) and the [contribute section](https://docs.duckcode.ai/contribute/repo-layout) of the docs site. Bugs and feature requests: [open an issue](https://github.com/duckcode-ai/dql/issues). ## License -Apache-2.0 — see [LICENSE](./LICENSE). +[Apache-2.0](./LICENSE). diff --git a/apps/docs/pages/guides/_meta.json b/apps/docs/pages/guides/_meta.json index 3426a127..5c07dea9 100644 --- a/apps/docs/pages/guides/_meta.json +++ b/apps/docs/pages/guides/_meta.json @@ -5,5 +5,6 @@ "dashboards": "Build a dashboard", "versioning": "Version & diff notebooks", "jaffle-shop": "Jaffle Shop walkthrough", - "migrate": "Migrate from Metabase / Looker / Hex" + "migrate": "Migrate from Metabase / Looker / Hex", + "faq": "FAQ" } diff --git a/apps/docs/pages/guides/faq.mdx b/apps/docs/pages/guides/faq.mdx new file mode 100644 index 00000000..85b48d45 --- /dev/null +++ b/apps/docs/pages/guides/faq.mdx @@ -0,0 +1,69 @@ +--- +title: FAQ +description: Frequently asked questions about DQL. +--- + +# FAQ + +## What is DQL? + +An open-source analytics language, CLI, and notebook. Every analytics answer +— SQL, chart config, owner, tests, parameters — lives in a single `.dql` +file in git. See [Concepts](/get-started/concepts). + +## Do I need a cloud warehouse to try it? + +No. DQL ships with DuckDB. Drop a CSV into `data/`, open `dql notebook`, and +query it immediately. No credentials, no configuration. + +## Do I need dbt? + +No. DQL works standalone. But if you already use dbt, `create-dql-app` +auto-detects a sibling dbt project and `dql sync dbt` imports your manifest, +metrics, and lineage. See [Import a dbt project](/guides/import-dbt). + +## How is this different from a Jupyter notebook? + +| | Jupyter | DQL notebook | +| ----------------- | -------------- | --------------------- | +| Primary language | Python | SQL | +| Runtime | Python kernel | DuckDB (or your warehouse) | +| Chart setup | matplotlib/Plotly code | declarative, auto-rendered | +| File format | `.ipynb` JSON | `.dqlnb` JSON | +| Git diff | noisy (outputs embedded) | clean (outputs sidecar) | + +## How is this different from Metabase, Looker, or Hex? + +Those are BI tools — queries live in a database or cloud service. DQL keeps +every artifact in git. You get code review, diff, branching, and blame on +your analytics. See [Migrate from Metabase / Looker / Hex](/guides/migrate). + +## What's the difference between `custom` and `semantic` blocks? + +- **`type = "custom"`** — the block's SQL executes directly against a connection +- **`type = "semantic"`** — the block references a semantic-layer metric; SQL is generated + +Start with `custom` blocks if you're new. + +## Can I use DQL from code, not just the CLI? + +Yes. The useful entry points: + +- `@duckcodeailabs/dql-core` — parser, AST, formatter, semantic analysis +- `@duckcodeailabs/dql-compiler` — IR lowering, codegen +- `@duckcodeailabs/dql-runtime` — browser runtime +- `@duckcodeailabs/dql-connectors` — database drivers + +## Does `dql test` hit a real database? + +Yes — assertions run against the configured connection. With no connection +configured, `dql test` falls back to structural checks only. + +## What's **not** in the open-source repo? + +Hosted workspaces, column-level lineage, scheduled runs, alerting, regulatory +governance packs, and AI/agentic block generation. See [Compatibility](/reference/compatibility). + +## Where do I report bugs or request features? + +[github.com/duckcode-ai/dql/issues](https://github.com/duckcode-ai/dql/issues). diff --git a/apps/docs/pages/reference/_meta.json b/apps/docs/pages/reference/_meta.json index 330de3db..f01c44c4 100644 --- a/apps/docs/pages/reference/_meta.json +++ b/apps/docs/pages/reference/_meta.json @@ -3,5 +3,7 @@ "language": "DQL language", "connectors": "Connectors", "semantic-layer": "Semantic layer", - "file-formats": "File formats" + "file-formats": "File formats", + "project-layout": "Project layout", + "compatibility": "Compatibility" } diff --git a/apps/docs/pages/reference/compatibility.mdx b/apps/docs/pages/reference/compatibility.mdx new file mode 100644 index 00000000..323aa923 --- /dev/null +++ b/apps/docs/pages/reference/compatibility.mdx @@ -0,0 +1,51 @@ +--- +title: Compatibility +description: Runtime, platform, and scope support matrix for OSS DQL. +--- + +# Compatibility + +## Runtime + +- **Node.js** — 18, 20, or 22 (active LTS) +- **Package managers** — npm, pnpm, yarn all work for consuming published packages +- **OS** — macOS, Linux, Windows (WSL recommended on Windows for the Tauri desktop build) + +## Connectors + +All 15 drivers ship in `@duckcodeailabs/dql-connectors`. For first-run local +exploration, start with `duckdb` or `file`. See [Connectors](/reference/connectors). + +## Chart types + +`bar`, `line`, `scatter`, `donut`, `pie`, `area`, `heatmap`, `kpi`, `table`, +`histogram`, `grouped-bar`, `stacked-bar`, `waterfall`, `funnel`, `gauge`, +`geo`, `combo`, `boxplot`, `forecast`, `stacked-area`. + +## Editor support + +- **VS Code** — [`dql.dql-language-support`](https://marketplace.visualstudio.com/items?itemName=dql.dql-language-support) extension (syntax, snippets, LSP) +- **Any LSP client** — `@duckcodeailabs/dql-lsp` speaks standard LSP + +## What's in this repo + +- DQL language (parser, formatter, AST) +- CLI (`init`, `notebook`, `compile`, `sync`, `diff`, `preview`, `serve`, `certify`, `test`, `fmt`, `doctor`) +- Notebook UI (SQL + DQL cells, block studio, lineage panel, git panel) +- Semantic layer (local YAML + dbt import) +- Lineage DAG (table / block / notebook granularity) +- 15 database connectors +- Plugin API (custom connectors, charts, governance rules) +- OpenLineage export + +## What's **not** in this repo + +Lives in the closed DuckCode Cloud product: + +- Column-level lineage +- Hosted multi-user workspace (SSO, RBAC, audit log) +- Scheduled runs, alerting, orchestration +- Regulatory governance packs (SOX, HIPAA, GDPR) +- AI / agentic block generation +- Premium connectors (SAP, Oracle, mainframe) +- Federated multi-warehouse blocks diff --git a/apps/docs/pages/reference/file-formats.mdx b/apps/docs/pages/reference/file-formats.mdx index eb6b1278..9b267e9a 100644 --- a/apps/docs/pages/reference/file-formats.mdx +++ b/apps/docs/pages/reference/file-formats.mdx @@ -1,6 +1,6 @@ --- title: File formats -description: .dql, .dqlnb, .run.json, and cdql.yaml. +description: .dql, .dqlnb, .run.json, and dql.config.json. --- # File formats @@ -31,11 +31,21 @@ block revenue_by_segment { - **Canonical whitespace** — `dql fmt` enforces it. CI should run `dql fmt --check .` - **Forward compatibility** — unknown top-level keys are preserved verbatim -## `.dqlnb` — notebook bundle (legacy) +## `.dqlnb` — notebook -Multi-cell notebooks shipped a bundled `.dqlnb` in early versions. From -v0.11 onwards, notebooks are plain `.dql` files with multiple cells. The -`.dqlnb` parser remains for backward compatibility. +JSON document holding a sequence of cells (`markdown`, `sql`, `dql`, `chart`). +Written in a stable key-order so git diffs stay readable. + +```json +{ + "version": 1, + "metadata": { "name": "Welcome", "createdAt": "2026-04-17T00:00:00Z" }, + "cells": [ + { "id": "c1", "type": "markdown", "source": "# Welcome" }, + { "id": "c2", "type": "sql", "source": "select 1" } + ] +} +``` ## `.run.json` — run snapshots @@ -61,30 +71,34 @@ Sibling file next to a notebook holding the last execution's results: - **Used for rehydration** — re-opening a notebook loads results without re-running queries - **Save trigger** — debounced 600ms after a cell's execution count / row count changes -## `cdql.yaml` — project config - -Lives at the project root. +## `dql.config.json` — project config -```yaml -project: - name: jaffle-analytics - version: 1 +Lives at the project root. Written by `dql init` and `create-dql-app`. -connections: - default: - driver: duckdb - path: ./warehouse.duckdb +```json +{ + "project": "jaffle-analytics", + "connections": { + "default": { "driver": "duckdb", "filepath": ":memory:" } + }, + "semanticLayer": { "provider": "dql", "path": "semantic-layer" }, + "dbt": { + "projectDir": "../dbt", + "manifestPath": "target/manifest.json" + }, + "governance": { + "required_fields": ["domain", "owner", "description"] + } +} +``` -dbt: - projectDir: ../jaffle-shop-dbt - include: - anchors: ["tag:core"] - maxDbtHops: 2 +Fields: -governance: - required_fields: [domain, owner, description] - certification_gates: [row_count_gt_zero, unique_pk] -``` +- **`project`** — human-readable project name +- **`connections`** — named database connections. `default` is used unless a block overrides it +- **`semanticLayer`** — `{ provider, path }`. Providers: `dql` (local YAML), `dbt`, `cubejs`, `snowflake` +- **`dbt`** — `{ projectDir, manifestPath }` for sibling dbt projects. `dql sync dbt` reads from here +- **`governance`** — required fields every block must declare to pass `dql certify` ## `dql-manifest.json` — build output diff --git a/apps/docs/pages/reference/project-layout.mdx b/apps/docs/pages/reference/project-layout.mdx new file mode 100644 index 00000000..fff83a64 --- /dev/null +++ b/apps/docs/pages/reference/project-layout.mdx @@ -0,0 +1,62 @@ +--- +title: Project layout +description: Directory conventions for a DQL project. +--- + +# Project layout + +A DQL project is a git repo with a few conventional directories. Everything +is plain files — no database, no lockfile, no hidden state. + +```text +my-dql-project/ +├─ dql.config.json # connection, semantic layer, dbt wiring +├─ package.json # npm scripts: notebook, compile, sync, doctor +├─ blocks/ # certified reusable .dql files +│ └─ revenue_by_segment.dql +├─ notebooks/ # .dqlnb interactive notebooks +│ └─ welcome.dqlnb +├─ semantic-layer/ # metrics, dimensions, hierarchies, cubes (YAML) +│ ├─ metrics/revenue.yaml +│ └─ dimensions/customer.yaml +├─ dashboards/ # composable dashboards (.dql dashboard blocks) +│ └─ overview.dql +├─ data/ # local CSV/Parquet — git-ignored by default +└─ .dql/ # manifest cache — git-ignored +``` + +## What each directory holds + +- **`blocks/`** — one `.dql` file per block. Governance fields (`domain`, + `owner`) are required by default; the certification check runs on CI. +- **`notebooks/`** — interactive analysis. Saved results live beside the + notebook as `.run.json` (git-ignored). +- **`semantic-layer/`** — metrics and dimensions authored locally. When you + run `dql sync dbt`, entries imported from a sibling dbt project are merged + here too. +- **`dashboards/`** — composed views that reference blocks. +- **`data/`** — sample data for local exploration. Production projects usually + query a warehouse instead. + +## Working with a sibling dbt project + +The default layout assumes your dbt project lives alongside your DQL project: + +```text +my-team/ +├─ dbt/ # your dbt project (dbt_project.yml) +│ └─ target/manifest.json # produced by `dbt parse` or `dbt build` +└─ dql/ # your DQL project + └─ dql.config.json # dbt.projectDir: "../dbt" +``` + +`create-dql-app` and `dql init` both auto-detect the sibling and wire the +config for you. Then `dql sync dbt` imports the manifest on demand. + +## What gets committed + +**Commit:** `blocks/`, `notebooks/`, `semantic-layer/`, `dashboards/`, +`dql.config.json`, `package.json`. + +**Don't commit:** `data/`, `.dql/`, `*.run.json`, `dql-manifest.json` (build +output). The default `.gitignore` from the scaffolder handles this. diff --git a/docs/01-start-here/README.md b/docs/01-start-here/README.md deleted file mode 100644 index 8257ec35..00000000 --- a/docs/01-start-here/README.md +++ /dev/null @@ -1,28 +0,0 @@ -# Start Here - -If you have never used DQL before, do not jump straight into the full reference. Follow this path in order. - -## Recommended Order - -1. [Install and First Project](./install-and-first-project.md) -2. [Run From Source](./run-from-source.md) if you are testing the repo locally before publishing -3. [dbt + Jaffle Shop Walkthrough](./dbt-jaffle-shop.md) if you want the dbt-backed semantic-layer path -4. [Core Workflows Overview](../02-core-workflows/README.md) - -## Choose Your Path - -### I just want to try DQL quickly - -Start with [Install and First Project](./install-and-first-project.md). - -### I want to test the repo locally from source - -Start with [Run From Source](./run-from-source.md). - -### I use dbt and want the semantic-layer workflow - -Start with [dbt + Jaffle Shop Walkthrough](./dbt-jaffle-shop.md). - -### I want to understand what DQL is before installing - -Read [Why DQL](../why-dql.md). diff --git a/docs/01-start-here/dbt-jaffle-shop.md b/docs/01-start-here/dbt-jaffle-shop.md deleted file mode 100644 index b1ad18c3..00000000 --- a/docs/01-start-here/dbt-jaffle-shop.md +++ /dev/null @@ -1,113 +0,0 @@ -# dbt + Jaffle Shop Walkthrough - -Use this path if you want to learn DQL with a dbt project that already includes semantic models and metrics. - -## What You Will Do - -1. Clone the Jaffle Shop semantic-layer course repo -2. Build it with dbt + DuckDB -3. Initialize DQL on top of that repo -4. Import the semantic layer -5. Open the notebook and lineage views - -## Prerequisites - -- Python 3.9+ -- Node.js 18, 20, or 22 -- Git - -## Step 1: Clone The Course Project - -```bash -git clone https://github.com/dbt-labs/Semantic-Layer-Online-Course.git jaffle-shop -cd jaffle-shop -``` - -## Step 2: Install dbt - -```bash -pip install dbt-duckdb -``` - -## Step 3: Create A Local DuckDB Profile - -```bash -cat > profiles.yml << 'EOF' -jaffle_shop: - target: dev - outputs: - dev: - type: duckdb - path: ./jaffle_shop.duckdb - schema: main - threads: 4 -EOF -``` - -## Step 4: Build The dbt Project - -```bash -dbt deps -dbt build --profiles-dir . -``` - -## Step 5: Install DQL And Initialize The Repo - -```bash -npm install -g @duckcodeailabs/dql-cli -dql init . -``` - -## Step 6: Import The dbt Semantic Layer - -```bash -dql semantic import dbt . -``` - -## Step 7: Verify The Project - -```bash -dql doctor -``` - -## Step 8: Open The Notebook - -```bash -dql notebook -``` - -## Step 9: Copy The Demo Lineage Notebook - -From the DQL repo, copy the demo notebook into your dbt project: - -```bash -mkdir -p notebooks -cp /path/to/DQL/dql/docs/examples/jaffle-shop-lineage-demo.dqlnb notebooks/ -``` - -This notebook is designed to show: - -- dbt model lineage upstream -- DQL block lineage in the middle -- notebook consumption downstream -- domain boundaries across customer, finance, and executive flows - -## Step 10: Import dbt Lineage - -```bash -dql compile --dbt-manifest target/manifest.json -dql lineage -``` - -## Step 11: Verify In The UI - -1. Open the `Lineage` sidebar and search for `revenue_by_customer_type` -2. Open `executive_revenue_rollup` in Block Studio -3. Use the `Lineage` tab in Block Studio to inspect the path summary -4. Open the fullscreen graph to inspect the broader dbt + DQL + notebook flow - -## What To Read Next - -- [Semantic Layer Workflow](../02-core-workflows/semantic-layer-workflow.md) -- [Lineage Workflow](../02-core-workflows/lineage-workflow.md) -- [Full detailed getting started guide](../getting-started.md) diff --git a/docs/01-start-here/install-and-first-project.md b/docs/01-start-here/install-and-first-project.md deleted file mode 100644 index 258a9040..00000000 --- a/docs/01-start-here/install-and-first-project.md +++ /dev/null @@ -1,62 +0,0 @@ -# Install And First Project - -This is the cleanest path for a brand-new user who knows nothing about DQL. - -## What You Will Do - -1. Install the CLI -2. Create your first DQL project -3. Verify the setup -4. Open the notebook - -## Prerequisites - -- Node.js 18, 20, or 22 -- npm - -## Step 1: Install The CLI - -```bash -npm install -g @duckcodeailabs/dql-cli -``` - -Check that it works: - -```bash -dql --help -``` - -## Step 2: Create A Project - -```bash -dql init my-dql-project -cd my-dql-project -``` - -This creates the basic DQL project structure for blocks, notebooks, and configuration. - -## Step 3: Verify The Project - -```bash -dql doctor -``` - -You should see that the project configuration and local runtime are valid. - -## Step 4: Open The Notebook - -```bash -dql notebook -``` - -This opens the DQL notebook locally in your browser. - -## What To Learn Next - -- [Notebook Workflow](../02-core-workflows/notebook-workflow.md) -- [Block Authoring Workflow](../02-core-workflows/block-authoring-workflow.md) -- [Semantic Layer Workflow](../02-core-workflows/semantic-layer-workflow.md) - -## If You Want A dbt Example - -Use the full [dbt + Jaffle Shop Walkthrough](./dbt-jaffle-shop.md). diff --git a/docs/01-start-here/run-from-source.md b/docs/01-start-here/run-from-source.md deleted file mode 100644 index 73ef8c78..00000000 --- a/docs/01-start-here/run-from-source.md +++ /dev/null @@ -1,93 +0,0 @@ -# Run DQL From Source - -Use this guide if you want to download the repo, install dependencies, run DQL locally, and test it before publishing to npm. - -## What You Will Do - -1. Clone the repo -2. Install dependencies -3. Build and test the monorepo -4. Run the CLI from source -5. Smoke-test the notebook and block workflows - -## Prerequisites - -- Node.js 18, 20, or 22 -- `pnpm` 9+ -- Git - -## Step 1: Clone The Repo - -```bash -git clone https://github.com/duckcode-ai/dql.git -cd dql -``` - -## Step 2: Install Dependencies - -```bash -pnpm install -``` - -## Step 3: Build The Repo - -```bash -pnpm build -``` - -## Step 4: Run The Test Suite - -```bash -pnpm test -``` - -## Step 5: Verify The CLI Entry Point - -```bash -pnpm --filter @duckcodeailabs/dql-cli exec dql --help -``` - -This confirms the CLI is runnable from the source checkout without needing a global npm install. - -## Step 6: Smoke-Test A New Project - -```bash -rm -rf /tmp/dql-smoke -pnpm --filter @duckcodeailabs/dql-cli exec dql init /tmp/dql-smoke -pnpm --filter @duckcodeailabs/dql-cli exec dql doctor /tmp/dql-smoke -pnpm --filter @duckcodeailabs/dql-cli exec dql notebook /tmp/dql-smoke -``` - -## Step 7: Smoke-Test A Block Workflow - -```bash -cd /tmp/dql-smoke -pnpm --filter @duckcodeailabs/dql-cli exec dql new block "Test Block" --domain test -pnpm --filter @duckcodeailabs/dql-cli exec dql parse blocks/test_block.dql -pnpm --filter @duckcodeailabs/dql-cli exec dql certify blocks/test_block.dql -pnpm --filter @duckcodeailabs/dql-cli exec dql build blocks/test_block.dql -``` - -## Step 8: Optional npm Package Check - -If you want to inspect the published CLI package contents before release: - -```bash -cd apps/cli -pnpm build -npm pack -``` - -## Recommended Manual UI Checks - -- open the notebook and confirm the welcome content loads -- run a SQL cell and confirm rows return -- open Block Studio and confirm the editor and results pane behave correctly -- open the connections screen and confirm driver settings render -- save a connection and test it - -## Related Docs - -- [Repo Testing](../repo-testing.md) -- [Publishing](../publishing.md) -- [OSS Readiness Checklist](../oss-readiness-checklist.md) diff --git a/docs/02-core-workflows/README.md b/docs/02-core-workflows/README.md deleted file mode 100644 index a0ce2dd5..00000000 --- a/docs/02-core-workflows/README.md +++ /dev/null @@ -1,20 +0,0 @@ -# Core Workflows - -After installation, these are the main workflows a user needs to understand. - -## Recommended Order - -1. [Notebook Workflow](./notebook-workflow.md) -2. [Block Authoring Workflow](./block-authoring-workflow.md) -3. [Semantic Layer Workflow](./semantic-layer-workflow.md) -4. [Lineage Workflow](./lineage-workflow.md) -5. [Demo Assets](../examples/README.md) - -## Goal Of This Section - -These pages explain how to use DQL as a product: - -- where to start in the UI -- what commands to run -- which file or page to use for each task -- what to read next when you want more detail diff --git a/docs/02-core-workflows/block-authoring-workflow.md b/docs/02-core-workflows/block-authoring-workflow.md deleted file mode 100644 index 8772b057..00000000 --- a/docs/02-core-workflows/block-authoring-workflow.md +++ /dev/null @@ -1,34 +0,0 @@ -# Block Authoring Workflow - -Blocks are the reusable answer units in DQL. - -## Typical Flow - -1. Create a block -2. Add SQL or semantic configuration -3. Add visualization settings -4. Add tests and ownership metadata -5. Validate and preview the block -6. Build or serve the result - -## Commands - -```bash -dql new block "Pipeline Health" -dql parse blocks/pipeline_health.dql -dql certify blocks/pipeline_health.dql -dql preview blocks/pipeline_health.dql --open -dql build blocks/pipeline_health.dql -``` - -## Use Blocks For - -- reusable analytics answers -- governed SQL -- semantic metric views -- certified visual outputs - -## Read Next - -- [Authoring Blocks](../authoring-blocks.md) -- [DQL Language Spec](../dql-language-spec.md) diff --git a/docs/02-core-workflows/lineage-workflow.md b/docs/02-core-workflows/lineage-workflow.md deleted file mode 100644 index 4edf8f79..00000000 --- a/docs/02-core-workflows/lineage-workflow.md +++ /dev/null @@ -1,96 +0,0 @@ -# Lineage Workflow - -Use lineage when you need to explain where a DQL answer came from, what it depends on, and which notebooks or downstream analytics will be affected by a change. - -For the open-source product, the core lineage story is: - -- source tables -- dbt sources and dbt models -- DQL blocks -- notebooks -- business domains and cross-domain flow - -## Recommended Demo Path - -Use Jaffle Shop and the ready-made notebook asset: - -- [dbt + Jaffle Shop Walkthrough](../01-start-here/dbt-jaffle-shop.md) -- [Jaffle Shop Lineage Demo notebook](../examples/jaffle-shop-lineage-demo.dqlnb) - -Copy the demo notebook into your project: - -```bash -mkdir -p notebooks -cp docs/examples/jaffle-shop-lineage-demo.dqlnb notebooks/ -``` - -## Typical Flow - -1. Build the dbt project so `target/manifest.json` exists -2. Compile DQL with dbt lineage imported -3. Open the notebook UI -4. Search for a block or notebook in the Lineage sidebar -5. Open a block in Block Studio and inspect the `Lineage` tab -6. Use CLI focus/search commands when you want a terminal view or JSON output - -## Commands - -```bash -dql compile --dbt-manifest target/manifest.json - -dql lineage -dql lineage --search revenue -dql lineage --focus revenue_by_customer_type -dql lineage --dashboard "Jaffle Shop Lineage Demo" -dql lineage --dbt -dql lineage --impact revenue_by_customer_type -``` - -## UI Walkthrough - -### Notebook Lineage Sidebar - -Use the sidebar when you want to move quickly through the graph: - -1. Open `Lineage` -2. Search for a source table, dbt model, DQL block, or notebook -3. Click a result to open a focused subgraph -4. Use `Open Graph View` for the fullscreen DAG - -### Fullscreen Lineage Graph - -Use the fullscreen graph when you want the broader path: - -- filter to source tables, dbt models, DQL blocks, notebooks, or domains -- focus on one node instead of showing the whole project -- inspect cross-domain flow visually - -### Block Studio Lineage Tab - -Use Block Studio when you are editing one block and need immediate context: - -- the active block auto-focuses lineage -- `Path Summary` shows upstream provenance and downstream consumption -- `Source to Block` helps explain dbt and table lineage -- `Block to Consumption` helps explain notebook and analytics usage - -## What Good Open-Source Lineage Looks Like - -For a dbt-backed DQL project, users should be able to read a chain like: - -```text -dbt source -> dbt model -> dbt model -> DQL block -> notebook -``` - -Or across domains: - -```text -customer domain -> finance domain -> executive domain -``` - -That is the core open-source adoption story. Column lineage, cross-tool SaaS connectors, and AI lineage queries remain out of scope for OSS. - -## Read Next - -- [Lineage Guide](../lineage.md) -- [Block Authoring Workflow](./block-authoring-workflow.md) diff --git a/docs/02-core-workflows/notebook-workflow.md b/docs/02-core-workflows/notebook-workflow.md deleted file mode 100644 index a64c917c..00000000 --- a/docs/02-core-workflows/notebook-workflow.md +++ /dev/null @@ -1,31 +0,0 @@ -# Notebook Workflow - -The notebook is the main working surface for ad hoc analysis and governed authoring. - -## Typical Flow - -1. Open the notebook with `dql notebook` -2. Browse files, blocks, or semantic assets -3. Run SQL or DQL content -4. Inspect results and charts -5. Save notebooks or convert logic into reusable blocks - -## Commands - -```bash -dql notebook -dql new notebook "Revenue Analysis" -``` - -## Use The Notebook For - -- SQL exploration -- parameterized analysis -- block editing -- chart previews -- semantic browsing - -## Read Next - -- [Notebook Guide](../notebook.md) -- [Block Authoring Workflow](./block-authoring-workflow.md) diff --git a/docs/02-core-workflows/semantic-layer-workflow.md b/docs/02-core-workflows/semantic-layer-workflow.md deleted file mode 100644 index 5233acef..00000000 --- a/docs/02-core-workflows/semantic-layer-workflow.md +++ /dev/null @@ -1,30 +0,0 @@ -# Semantic Layer Workflow - -Use the semantic layer when you want governed metrics and dimensions instead of rewriting logic in every query. - -## Typical Flow - -1. Choose the provider mode -2. Import or define semantic assets -3. Validate the semantic catalog -4. Browse metrics and dimensions in the notebook -5. Compose blocks and queries from semantic selections - -## Provider Modes - -- `dql` for local YAML assets -- `dbt` for dbt semantic models -- `cubejs` for Cube.js models -- `snowflake` for Snowflake semantic views - -## Common Commands - -```bash -dql semantic import dbt . -dql doctor -``` - -## Read Next - -- [Semantic Layer Guide](../semantic-layer-guide.md) -- [dbt + Jaffle Shop Walkthrough](../01-start-here/dbt-jaffle-shop.md) diff --git a/docs/03-guides/README.md b/docs/03-guides/README.md deleted file mode 100644 index bd615aa1..00000000 --- a/docs/03-guides/README.md +++ /dev/null @@ -1,13 +0,0 @@ -# Guides - -These guides are for users who already know the main flow and want focused help on a specific topic. - -## Topics - -- [Data Sources](../data-sources.md) -- [Project Structure](../project-structure.md) -- [Project Config](../project-config.md) -- [Examples](../examples.md) -- [Use Cases](../use-cases.md) -- [FAQ](../faq.md) -- [Why DQL](../why-dql.md) diff --git a/docs/04-reference/README.md b/docs/04-reference/README.md deleted file mode 100644 index 1e964a57..00000000 --- a/docs/04-reference/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Reference - -Use this section when you already know what you are trying to do and need exact syntax, commands, or support boundaries. - -## Reference Docs - -- [CLI Reference](../cli-reference.md) -- [DQL Language Spec](../dql-language-spec.md) -- [Compatibility](../compatibility.md) -- [Project Config](../project-config.md) -- [V1 Support Scope](../v1-support-scope.md) diff --git a/docs/05-maintainers/README.md b/docs/05-maintainers/README.md deleted file mode 100644 index 2760adb0..00000000 --- a/docs/05-maintainers/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# Maintainer Docs - -This section is for contributors, release owners, and anyone validating the repo before publish. - -## Recommended Order - -1. [Run From Source](../01-start-here/run-from-source.md) -2. [Repo Testing](../repo-testing.md) -3. [Publishing](../publishing.md) -4. [OSS Readiness Checklist](../oss-readiness-checklist.md) - -## Use This Section For - -- local source validation -- release confidence checks -- publish steps -- open-source readiness review diff --git a/docs/README.md b/docs/README.md index cc8624a7..6e226d39 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,30 +1,17 @@ # docs/ -**The user-facing docs have moved to [`apps/docs/`](../apps/docs/) (Nextra). -Site: [docs.duckcode.ai](https://docs.duckcode.ai).** +**User-facing documentation lives at [docs.duckcode.ai](https://docs.duckcode.ai)** +(source in [`apps/docs/`](../apps/docs/)). -This folder now holds only: +This folder only holds internal maintainer notes that don't belong on the +public site: -- **Long-form references** not yet ported — e.g. [`dql-language-spec.md`](./dql-language-spec.md) -- **Maintainer / internal docs** — [`publishing.md`](./publishing.md), [`repo-testing.md`](./repo-testing.md), [`oss-readiness-checklist.md`](./oss-readiness-checklist.md), [`v1-support-scope.md`](./v1-support-scope.md) -- **Migration notes** — [`migration-guides/`](./migration-guides/) -- **Example notebooks** — [`examples/`](./examples/) +- [`oss-readiness-checklist.md`](./oss-readiness-checklist.md) — pre-release + OSS readiness checklist -New documentation goes in `apps/docs/pages/`, not here. - -## Canonical IA (in `apps/docs`) - -``` -Get Started /get-started/ install · quickstart · concepts -Guides /guides/ task-oriented walkthroughs -Reference /reference/ CLI · language · connectors · file formats · semantic layer -Architecture /architecture/ overview · dbt-integration · lineage-model · plugin-api · openlineage -Contribute /contribute/ repo-layout · testing · releasing -``` - -## Running the site locally +Everything else — quickstart, guides, reference, architecture, contributing — +is in Nextra at `apps/docs/pages/`. To preview the site locally: ```bash -pnpm -F @duckcodeailabs/docs dev -# http://localhost:3030 +cd apps/docs && pnpm dev ``` diff --git a/docs/compatibility.md b/docs/compatibility.md deleted file mode 100644 index 09c71487..00000000 --- a/docs/compatibility.md +++ /dev/null @@ -1,183 +0,0 @@ -# Compatibility - -This document describes the current DQL open-source compatibility target for local use, package consumption, browser notebook workflows, and experimentation. - -## Runtime Requirements - -### Node.js - -- Node.js 18 or newer is required for the public CLI and package workspace. - -### Package Managers - -- `npm` works for published package installation. -- `pnpm` is recommended when working from source in the monorepo. - -## Supported CLI Workflows - -The current open-source Phase 1 workflow centers on: - -- `dql init` -- `dql notebook` -- `dql doctor` -- `dql parse` -- `dql fmt` -- `dql info` -- `dql test` -- `dql certify` -- `dql preview` -- `dql build` -- `dql serve` - -Recommended first-run path: - -```bash -dql init my-dql-project -cd my-dql-project -dql doctor -dql new block "Pipeline Health" -dql preview blocks/pipeline_health.dql --open -``` - -## Supported Project Config - -Current local-first docs and starter templates assume `dql.config.json` as the project config file. - -It is used for: - -- default connection settings -- preview host and port -- auto-open behavior -- theme selection - -## Connector Compatibility - -The connector layer currently includes these drivers: - -| Driver | Status | Best Use | -|---|---|---| -| `file` | Recommended for first-run | Local CSV/Parquet exploration | -| `duckdb` | Recommended for first-run | Local DuckDB-backed analytics | -| `sqlite` | Supported | Lightweight local database workflows | -| `postgresql` | Supported | Warehouse or OLTP-backed query execution | -| `mysql` | Supported | MySQL-backed query execution | -| `mssql` | Supported | SQL Server-backed query execution | -| `bigquery` | Supported | BigQuery-backed query execution | -| `snowflake` | Supported | Snowflake-backed query execution | - -### Recommendation - -For open-source evaluation, start with: - -- `file` -- `duckdb` - -These have the lowest onboarding friction and work best with the starter project model. - -## Visualization Compatibility - -The language spec currently documents support for these chart types: - -- `bar` -- `line` -- `scatter` -- `donut` -- `pie` -- `area` -- `heatmap` -- `kpi` -- `table` -- `histogram` -- `grouped-bar` -- `stacked-bar` -- `waterfall` -- `funnel` -- `gauge` -- `geo` -- `combo` -- `boxplot` -- `forecast` -- `stacked-area` - -For easiest first-run results, start with: - -- `bar` -- `line` -- `table` -- `kpi` - -## Language Surface Compatibility - -### Reusable blocks - -Supported block types: - -- `type = "custom"` -- `type = "semantic"` - -### Dashboards and workbooks - -DQL also supports: - -- `dashboard "..." { ... }` -- `workbook "..." { ... }` - -### Recommended starting point - -For open-source adoption and local experimentation, start with: - -- reusable `custom` blocks -- local data -- `dql preview` - -## Editor Compatibility - -The open DQL repo includes: - -- a VS Code extension -- a reusable language-server package: `@duckcodeailabs/dql-lsp` - -These provide: - -- syntax highlighting -- formatting -- hover docs -- completions -- diagnostics - -## Platform Notes - -Because the connector workspace depends on several database client libraries, some setups may require compatible native binaries or system libraries depending on the driver you use. - -If you want the lowest-friction setup, prefer the local-first path: - -- starter project -- `dql notebook` -- `file` or `duckdb` connector -- local preview server - -## Known OSS Phase 1 Limits - -- The easiest supported path today is local-first preview and browser notebook exploration, not full enterprise deployment. -- `dql preview` and `dql serve` use a lightweight local query API for interactive rendering. -- `dql notebook` uses the same local query API and adds notebook execution plus connection forms. -- `dql test` is most useful when a runnable query execution path is configured. -- The fastest adoption path is currently the CLI and source workspace; published package availability may depend on your release process. - -## Recommended Baseline for New Users - -If you want the smoothest experience, use this baseline: - -- Node.js 18+ -- starter project created with `dql init` -- local sample data -- `file` or `duckdb` connection -- `dql preview` for experimentation - -## Related Docs - -- [Getting Started](./getting-started.md) -- [FAQ](./faq.md) -- [Project Config](./project-config.md) -- [Data Sources](./data-sources.md) -- [CLI Reference](./cli-reference.md) diff --git a/docs/data-sources.md b/docs/data-sources.md deleted file mode 100644 index 5e1ca5e3..00000000 --- a/docs/data-sources.md +++ /dev/null @@ -1,546 +0,0 @@ -# Data Sources & Connector Reference - -DQL supports **14 database connectors** out of the box. Start with local file mode (no database needed), then connect to your warehouse when ready. - -> **Step-by-step walkthrough:** [Enterprise Getting Started](./enterprise-getting-started.md) shows how to configure your database connection, browse schema, and build blocks — all from the notebook UI. - ---- - -## Quick Start (No Database) - -```bash -npm install -g @duckcodeailabs/dql-cli -dql init my-project && cd my-project -dql notebook -``` - -The default `driver: "file"` uses DuckDB in-memory. Query local CSV, Parquet, and JSON files: - -```sql -SELECT * FROM read_csv_auto('./data/orders.csv') -SELECT * FROM read_parquet('./data/events.parquet') -SELECT * FROM read_json('./data/config.json') -``` - ---- - -## Connector Config Reference - -All connections go in `dql.config.json` under `defaultConnection`. Use `${ENV_VAR}` for secrets — DQL resolves them from your shell environment. - -### Local / Embedded - -#### DuckDB In-Memory (`file`) - -```json -{ - "defaultConnection": { - "driver": "file", - "filepath": ":memory:" - } -} -``` - -| Field | Required | Description | -|---|---|---| -| `driver` | | Must be `"file"` | -| `filepath` | | Path to `.duckdb` file, or `":memory:"` (default) | - -#### DuckDB File (`duckdb`) - -```json -{ - "defaultConnection": { - "driver": "duckdb", - "filepath": "./data/warehouse.duckdb" - } -} -``` - -| Field | Required | Description | -|---|---|---| -| `driver` | | Must be `"duckdb"` | -| `filepath` | | Path to the `.duckdb` database file | - -#### SQLite (`sqlite`) - -```json -{ - "defaultConnection": { - "driver": "sqlite", - "database": "./data/analytics.sqlite" - } -} -``` - -| Field | Required | Description | -|---|---|---| -| `driver` | | Must be `"sqlite"` | -| `database` | | Path to the `.sqlite` or `.db` file | - ---- - -### Relational Databases - -#### PostgreSQL (`postgres`) - -```json -{ - "defaultConnection": { - "driver": "postgres", - "host": "localhost", - "port": 5432, - "database": "analytics", - "username": "analyst", - "password": "${POSTGRES_PASSWORD}", - "ssl": false - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"postgres"` | -| `host` | | — | Hostname or IP address | -| `port` | | `5432` | Port number | -| `database` | | — | Database name | -| `username` | | — | Database user | -| `password` | | — | Password (use `${ENV_VAR}`) | -| `ssl` | | `false` | Enable SSL/TLS | -| `schema` | | `"public"` | Default schema | - -**Works with:** PostgreSQL, Supabase, Amazon RDS, Aurora, Neon, CockroachDB - -#### MySQL (`mysql`) - -```json -{ - "defaultConnection": { - "driver": "mysql", - "host": "localhost", - "port": 3306, - "database": "analytics", - "username": "root", - "password": "${MYSQL_PASSWORD}", - "ssl": false - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"mysql"` | -| `host` | | — | Hostname or IP address | -| `port` | | `3306` | Port number | -| `database` | | — | Database name | -| `username` | | — | Database user | -| `password` | | — | Password (use `${ENV_VAR}`) | -| `ssl` | | `false` | Enable SSL/TLS | - -**Works with:** MySQL, MariaDB, PlanetScale, TiDB, Vitess - -#### SQL Server (`mssql`) - -```json -{ - "defaultConnection": { - "driver": "mssql", - "host": "localhost", - "port": 1433, - "database": "analytics", - "username": "sa", - "password": "${MSSQL_PASSWORD}", - "ssl": false - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"mssql"` | -| `host` | | — | Hostname or IP address | -| `port` | | `1433` | Port number | -| `database` | | — | Database name | -| `username` | | — | Database user | -| `password` | | — | Password (use `${ENV_VAR}`) | -| `ssl` | | `false` | Enable SSL/TLS | - -**Works with:** SQL Server, Azure SQL Database, Azure SQL Managed Instance - ---- - -### Cloud Data Warehouses - -#### Snowflake (`snowflake`) - -```json -{ - "defaultConnection": { - "driver": "snowflake", - "account": "your-account.snowflakecomputing.com", - "username": "your_user", - "password": "${SNOWFLAKE_PASSWORD}", - "database": "ANALYTICS", - "schema": "PUBLIC", - "warehouse": "COMPUTE_WH", - "role": "ANALYST" - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"snowflake"` | -| `account` | | — | Snowflake account identifier (e.g., `abc123.us-east-1`) | -| `username` | | — | Snowflake username | -| `password` | | — | Password (use `${ENV_VAR}`) | -| `database` | | — | Database name | -| `schema` | | `"PUBLIC"` | Default schema | -| `warehouse` | | — | Virtual warehouse name | -| `role` | | — | Role to use for the session | - -#### BigQuery (`bigquery`) - -```json -{ - "defaultConnection": { - "driver": "bigquery", - "project": "your-gcp-project-id", - "dataset": "analytics", - "keyFilename": "./service-account.json" - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"bigquery"` | -| `project` | | — | GCP project ID | -| `dataset` | | — | Default dataset | -| `keyFilename` | | — | Path to service account JSON key file | -| `location` | | `"US"` | Dataset location (e.g., `"EU"`, `"us-central1"`) | - -**Auth:** If `keyFilename` is not set, uses Application Default Credentials (`gcloud auth application-default login`). - -#### Amazon Redshift (`redshift`) - -```json -{ - "defaultConnection": { - "driver": "redshift", - "host": "cluster.abc123.us-east-1.redshift.amazonaws.com", - "port": 5439, - "database": "analytics", - "username": "analyst", - "password": "${REDSHIFT_PASSWORD}", - "ssl": true - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"redshift"` | -| `host` | | — | Cluster endpoint | -| `port` | | `5439` | Port number | -| `database` | | — | Database name | -| `username` | | — | Database user | -| `password` | | — | Password (use `${ENV_VAR}`) | -| `ssl` | | `true` | Enable SSL (recommended) | -| `schema` | | `"public"` | Default schema | - -#### Databricks SQL (`databricks`) - -```json -{ - "defaultConnection": { - "driver": "databricks", - "host": "dbc-example.cloud.databricks.com", - "warehouse": "your-sql-warehouse-id", - "catalog": "main", - "schema": "analytics", - "token": "${DATABRICKS_TOKEN}" - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"databricks"` | -| `host` | | — | Workspace hostname | -| `warehouse` | | — | SQL Warehouse ID (from the HTTP Path) | -| `catalog` | | `"main"` | Unity Catalog name | -| `schema` | | `"default"` | Default schema | -| `token` | | — | Personal access token (use `${ENV_VAR}`) | - -#### ClickHouse (`clickhouse`) - -```json -{ - "defaultConnection": { - "driver": "clickhouse", - "host": "play.clickhouse.com", - "port": 8443, - "database": "default", - "username": "play", - "password": "${CLICKHOUSE_PASSWORD}", - "ssl": true - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"clickhouse"` | -| `host` | | — | Hostname | -| `port` | | `8443` | HTTP(S) port | -| `database` | | `"default"` | Database name | -| `username` | | — | User | -| `password` | | — | Password (use `${ENV_VAR}`) | -| `ssl` | | `true` | Enable SSL | - -**Works with:** ClickHouse Cloud, self-hosted ClickHouse - -#### Amazon Athena (`athena`) - -```json -{ - "defaultConnection": { - "driver": "athena", - "region": "us-east-1", - "database": "analytics", - "outputLocation": "s3://my-query-results/", - "workgroup": "primary" - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"athena"` | -| `region` | | — | AWS region | -| `database` | | — | Athena database (Glue catalog) | -| `outputLocation` | | — | S3 path for query results | -| `workgroup` | | `"primary"` | Athena workgroup | - -**Auth:** Uses AWS SDK default credential chain (env vars, `~/.aws/credentials`, IAM role). - -#### Trino (`trino`) - -```json -{ - "defaultConnection": { - "driver": "trino", - "host": "trino.example.com", - "port": 8080, - "catalog": "lakehouse", - "schema": "analytics", - "username": "analyst", - "password": "${TRINO_PASSWORD}", - "ssl": true - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"trino"` | -| `host` | | — | Trino coordinator hostname | -| `port` | | `8080` | Port number | -| `catalog` | | — | Default catalog | -| `schema` | | `"default"` | Default schema | -| `username` | | — | User | -| `password` | | — | Password (if auth is enabled) | -| `ssl` | | `false` | Enable SSL | - -**Works with:** Trino, Starburst, Starburst Galaxy - -#### Microsoft Fabric (`fabric`) - -```json -{ - "defaultConnection": { - "driver": "fabric", - "host": "workspace.datawarehouse.fabric.microsoft.com", - "port": 1433, - "database": "analytics", - "username": "user", - "password": "${FABRIC_PASSWORD}", - "ssl": true - } -} -``` - -| Field | Required | Default | Description | -|---|---|---|---| -| `driver` | | — | Must be `"fabric"` | -| `host` | | — | Fabric SQL endpoint | -| `port` | | `1433` | Port number | -| `database` | | — | Lakehouse or warehouse name | -| `username` | | — | Azure AD user or service principal | -| `password` | | — | Password or token (use `${ENV_VAR}`) | -| `ssl` | | `true` | Enable SSL (required for Fabric) | - ---- - -## All Connectors at a Glance - -| Driver | `driver` value | Typical use | Auth method | -|---|---|---|---| -| DuckDB In-Memory | `file` | Local CSV/Parquet analysis | None | -| DuckDB File | `duckdb` | Persistent local warehouse | None | -| SQLite | `sqlite` | Lightweight embedded DB | None | -| PostgreSQL | `postgres` | Supabase, RDS, Aurora, Neon | user/password | -| MySQL | `mysql` | MariaDB, PlanetScale, TiDB | user/password | -| SQL Server | `mssql` | Azure SQL, on-prem MSSQL | user/password | -| Snowflake | `snowflake` | Cloud data warehouse | user/password + account | -| BigQuery | `bigquery` | Google Cloud analytics | Service account or ADC | -| Redshift | `redshift` | AWS data warehouse | user/password | -| Databricks | `databricks` | Lakehouse analytics | Personal access token | -| ClickHouse | `clickhouse` | Real-time analytics | user/password | -| Athena | `athena` | S3-based serverless queries | AWS credentials | -| Trino | `trino` | Federated queries, Starburst | user/password | -| Fabric | `fabric` | Microsoft Fabric lakehouse | Azure AD | - ---- - -## Security Best Practices - -**Never hardcode secrets.** Use environment variables: - -```json -{ - "defaultConnection": { - "driver": "snowflake", - "password": "${SNOWFLAKE_PASSWORD}" - } -} -``` - -Set the variable before running DQL: - -```bash -export SNOWFLAKE_PASSWORD="your-secret" -npx @duckcodeailabs/dql-cli notebook -``` - -Or use a `.env` file (add to `.gitignore`): - -```bash -# .env (never commit this) -SNOWFLAKE_PASSWORD=your-secret -POSTGRES_PASSWORD=your-secret -``` - ---- - -## Semantic Layer + Connector Integration - -All 14 connectors support the semantic layer. DQL generates **database-specific SQL** for each driver. - -### SQL dialect differences (handled automatically) - -| Feature | PostgreSQL | BigQuery | MySQL | Snowflake | MSSQL | ClickHouse | SQLite | -|---------|-----------|----------|-------|-----------|-------|------------|--------| -| DATE_TRUNC | `DATE_TRUNC('month', col)` | `DATE_TRUNC(col, MONTH)` | `DATE_FORMAT(col, '%Y-%m-01')` | `DATE_TRUNC('month', col)` | `DATETRUNC(month, col)` | `toStartOfMonth(col)` | `STRFTIME('%Y-%m-01', col)` | -| LIMIT | `LIMIT N` | `LIMIT N` | `LIMIT N` | `LIMIT N` | `OFFSET 0 ROWS FETCH NEXT N ROWS ONLY` | `LIMIT N` | `LIMIT N` | -| Identifier quoting | `"col"` | `` `col` `` | `` `col` `` | `"col"` | `[col]` | `"col"` | `"col"` | - -### Semantic Query API - -The notebook runtime exposes `POST /api/semantic-query`: - -```json -{ - "metrics": ["total_revenue"], - "dimensions": ["channel"], - "timeDimension": { "name": "order_date", "granularity": "month" }, - "filters": [{ "dimension": "channel", "operator": "equals", "values": ["web"] }], - "limit": 100 -} -``` - -Uses `defaultConnection` automatically. Override with a `"connection": { ... }` field if needed. - ---- - -## Test Your Connection - -### From the CLI - -```bash -dql doctor -``` - -Look for: -``` - ✓ Default connection driver=postgres - ✓ Local query runtime driver=postgres is available -``` - -### From the Notebook Connection Panel - -1. Launch: `dql notebook` -2. Click the **Connection** icon (plug icon) in the left sidebar -3. You'll see the current connection config and a status indicator -4. Click **Test Connection** to verify -5. See "Connected to postgres successfully" (or your driver) - -### Change connections at runtime - -The Connection Panel lets you switch databases without restarting: - -1. Select a **Quick Connect** preset (DuckDB, PostgreSQL, Snowflake) or fill in fields manually -2. Click **Save** — DQL hot-swaps the connection immediately -3. The Schema sidebar refreshes with the new database's tables -4. All subsequent queries run against the new connection - -> You can also edit `dql.config.json` directly and restart `dql notebook`. - ---- - -## Lineage with Remote Databases - -All 14 connectors work with DQL's lineage engine. When your blocks query remote tables, lineage automatically tracks the source tables: - -```dql -block "Revenue Summary" { - domain = "finance" - type = "custom" - owner = "finance-team" - query = """ - SELECT segment, SUM(amount) AS revenue - FROM analytics.public.fct_orders - GROUP BY segment - """ -} -``` - -`dql lineage` shows `analytics.public.fct_orders` as a `source_table` node connected to the block. Schema-qualified names are preserved. - -Use `ref()` to connect blocks that span different domains or teams: - -```dql -block "Exec Dashboard" { - domain = "executive" - query = """ - SELECT * FROM ref("revenue_summary") - """ -} -``` - -Cross-domain flows (e.g., `finance → executive`) are detected automatically. See the [Lineage Guide](./lineage.md) for the full lineage CLI and API. - ---- - -## Troubleshooting - -| Problem | Cause | Fix | -|---------|-------|-----| -| `command not found: dql` | CLI not installed | Use `npx @duckcodeailabs/dql-cli` | -| `read_csv_auto(...) file not found` | Wrong path | Use `./data/file.csv` (project-relative) | -| Connection refused | Database not running or wrong host/port | Check `host`, `port`, firewall rules | -| Authentication failed | Wrong credentials | Verify `username`, `password`, env vars | -| SSL required | Cloud database requires SSL | Add `"ssl": true` to config | -| `${ENV_VAR}` not resolved | Environment variable not set | `export ENV_VAR=value` before running | -| DuckDB native module error | Node version changed | Run `pnpm install` to rebuild bindings | diff --git a/docs/dql-language-spec.md b/docs/dql-language-spec.md deleted file mode 100644 index 35f28139..00000000 --- a/docs/dql-language-spec.md +++ /dev/null @@ -1,270 +0,0 @@ -# DQL Language Specification - -**Version:** 1.0.0 -**Status:** Implemented in `@duckcodeailabs/dql-core` - ---- - -## Overview - -DQL (Domain Query Language) is a declarative language for defining analytics blocks — self-contained, testable, version-controlled units of data analysis. Each block wraps SQL with metadata, optional visualization config, parameters, and test assertions. - -## Lexical Structure - -### Keywords - -``` -block, domain, type, description, tags, owner, params, query, visualization, -tests, assert, var, chart, import, use, dashboard, workbook, filter, layout -``` - -### Chart Types - -``` -bar, line, scatter, donut, pie, area, heatmap, kpi, table, histogram, -grouped-bar, stacked-bar, waterfall, funnel, gauge, geo, combo, boxplot, -forecast, stacked-area -``` - -### Literals - -- **Strings:** `"double quoted"` or `"""triple quoted for multi-line"""` -- **Numbers:** `42`, `3.14`, `-1` -- **Booleans:** `true`, `false` -- **Arrays:** `["a", "b", "c"]` - -### Comments - -```dql -// Single-line comment -``` - -### SQL Fragments - -SQL is embedded in triple-quoted strings. Parameter interpolation supports both `${param_name}` and `{param_name}` (legacy). - -## Block Declaration - -A block is the atomic unit of the platform: - -```dql -block "Block Name" { - domain = "domain_name" - type = "custom" // required: "custom" or "semantic" - description = "Human-readable description" - tags = ["tag1", "tag2"] - owner = "username" - - params { ... } - query = """...""" // required for type = "custom" - visualization { ... } - tests { ... } -} -``` - -### Block Types - -Every block must declare its execution type: - -| Value | Description | -|-------|-------------| -| `type = "custom"` | Certified SQL block — the `query` field contains the SQL executed at runtime | -| `type = "semantic"` | Semantic layer reference — the `metric` field references a dbt metric by name; SQL is managed by the semantic layer, not the block | - -```dql -// Custom block (certified SQL) -block "Revenue by Segment" { - type = "custom" - domain = "revenue" - query = """SELECT segment, SUM(revenue) FROM fct_revenue GROUP BY 1""" - // ... -} - -// Semantic block (dbt metric reference) -block "ARR Growth" { - type = "semantic" - domain = "finance" - metric = "annual_recurring_revenue" - // query field must be omitted -} -``` - -### Required Fields - -| Field | Type | Description | -|-------|------|-------------| -| `domain` | string | Business domain (revenue, retention, sales, etc.) | -| `type` | string | Block type — must be `"semantic"` or `"custom"` (see Block Types above) | - -### Optional Fields - -| Field | Type | Description | -|-------|------|-------------| -| `description` | string | Human-readable description | -| `tags` | string[] | Discoverability tags | -| `owner` | string | Block owner/maintainer | - -## Parameters - -Parameters allow blocks to be reusable with different inputs: - -```dql -params { - period = "current_quarter" - limit = 10 - include_forecast = false -} -``` - -Parameters are interpolated in SQL via `${param_name}` or `{param_name}`. - -## Query - -The query field contains SQL wrapped in triple-quoted strings: - -```dql -query = """ - SELECT segment, SUM(revenue) AS revenue - FROM fct_revenue - WHERE period = ${period} - GROUP BY segment - ORDER BY revenue DESC -""" -``` - -## Visualization - -The `visualization` section is optional. Query-only blocks can omit it when the asset is meant for reuse, validation, or downstream composition rather than direct chart rendering. - -Visualization config maps query columns to chart properties: - -```dql -visualization { - chart = "bar" - x = segment - y = revenue - color = "#7c8cf5" -} -``` - -### Chart Properties - -| Property | Description | -|----------|-------------| -| `x` | X-axis column | -| `y` | Y-axis column (or array for multi-series) | -| `color` | Color hex or column for color-by | -| `color_by` | Column for categorical coloring | -| `label` | Label column (for donut/funnel) | -| `value` | Value column (for donut/funnel/kpi) | -| `format` | Number format (compact, currency, percent) | - -## Test Assertions - -Blocks can include test assertions that validate query results: - -```dql -tests { - assert row_count > 0 - assert max(revenue) < 10000000 - assert min(retention_pct) >= 0 - assert churn_rate <= 1 -} -``` - -### Assertion Operators - -`>`, `<`, `>=`, `<=`, `==`, `!=` - -## Dashboard Declaration - -Multiple blocks can be composed into dashboards: - -```dql -dashboard "Revenue Overview" { - use RevenueBySegment - use "ARR Trend" - - chart.bar( - SELECT segment, SUM(revenue) AS revenue FROM fct_revenue GROUP BY segment, - x = segment, - y = revenue - ) -} -``` - -## Import System - -Blocks can import from other files: - -```dql -import { RevenueBySegment } from "./revenue/rev-by-segment.dql" -``` - -## Semantic Layer Integration - -Blocks can reference metrics and dimensions from the semantic layer. The semantic layer can also carry hierarchy definitions and optional companion business metadata for individual blocks: - -```yaml -# semantic-layer/metrics/revenue.yaml -metrics: - - name: total_revenue - sql: SUM(amount) - type: sum - table: fct_revenue -``` - -```yaml -# semantic-layer/hierarchies/revenue_time.yaml -name: revenue_time -label: Revenue Time -description: Calendar drill path for revenue reporting -domain: revenue -levels: - - name: fiscal_year - dimension: fiscal_year - order: 1 - - name: fiscal_quarter - dimension: fiscal_quarter - order: 2 -``` - -```yaml -# semantic-layer/blocks/revenue_by_segment.yaml -name: revenue_by_segment -block: revenue_by_segment -domain: revenue -description: Business context and semantic mappings for the segment revenue block -semanticMappings: - segment: segment_tier - revenue: total_revenue -reviewStatus: review -``` - -The `SemanticLayer` class validates metric, dimension, and hierarchy definitions. Companion metadata is a separate typed YAML contract for block-level business context. - -## AST Node Types - -The parser produces a typed AST with 30+ node kinds: - -- `ProgramNode` — root node -- `BlockDeclNode` — block declaration -- `DashboardDeclNode` — dashboard declaration -- `ChartDeclNode` — chart configuration -- `QueryNode` — SQL query -- `ParamDeclNode` — parameter declaration -- `TestAssertionNode` — test assertion -- `ImportDeclNode` — import declaration -- `UseDeclNode` — use declaration -- `FilterDeclNode` — filter declaration -- `LayoutDeclNode` — layout configuration - -## Compilation Pipeline - -``` -Source → Lexer → Tokens → Parser → AST → Semantic Analysis → IR → Code Generation - ├── Vega-Lite - ├── React (visx) - ├── HTML - └── Runtime JS -``` diff --git a/docs/examples/README.md b/docs/examples/README.md deleted file mode 100644 index b69f217a..00000000 --- a/docs/examples/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# Demo Assets - -Copy these files into a DQL project when you want a concrete product demo instead of starting from a blank notebook. - -## Available Assets - -- [Jaffle Shop Lineage Demo](./jaffle-shop-lineage-demo.dqlnb) - -## Recommended Usage - -1. Start with the [dbt + Jaffle Shop walkthrough](../01-start-here/dbt-jaffle-shop.md) -2. Copy the demo notebook into your project's `notebooks/` folder -3. Run `dql compile --dbt-manifest target/manifest.json` -4. Open `dql notebook` -5. Use the Lineage sidebar, fullscreen graph, and Block Studio lineage tab on the demo blocks diff --git a/docs/faq.md b/docs/faq.md deleted file mode 100644 index b24717ad..00000000 --- a/docs/faq.md +++ /dev/null @@ -1,176 +0,0 @@ -# FAQ - -## What is DQL? - -DQL is an open language for defining durable analytics assets in Git. A DQL block can combine metadata, SQL, parameters, visualization settings, and test assertions in one reusable file. - -## Does DQL require DuckCode Studio? - -No. DQL works standalone. - -You can scaffold a local project, validate blocks, preview charts, and build bundles directly with the DQL CLI. - -## Does DQL have a notebook UI? - -Yes. Run `dql notebook` from any DQL project root and it opens a local browser notebook. - -The notebook supports: -- **SQL cells** — run queries against local CSV/Parquet files via DuckDB -- **Markdown cells** — prose, headers, callouts -- **Param widgets** — adjust query parameters interactively without editing SQL -- **Auto-charting** — results render as charts based on visualization config in your `.dql` files -- **Export** — save results or share notebook files (`.dqlnb`) - -```bash -dql init . -dql notebook -``` - -## Can I use DQL for just the notebook, without writing .dql files? - -Yes. The notebook works with plain SQL cells. You do not need to author `.dql` block files to explore data. - -Drop a CSV in the `data/` folder of your project, open `dql notebook`, and start writing SQL. `.dql` block files are the durable output you create when you want to commit a query as a reusable, testable asset — but they are not required to use the notebook. - -## How is DQL different from a Jupyter notebook? - -Jupyter is Python-first and requires a kernel runtime. DQL notebook is SQL-first with no Python runtime needed. - -Key differences: - -| | Jupyter | DQL Notebook | -|---|---|---| -| Primary language | Python | SQL | -| Execution runtime | Python kernel | DuckDB (in-browser) | -| Chart setup | matplotlib / Plotly code | Declarative config, auto-rendered | -| File format | `.ipynb` JSON | `.dqlnb` JSON | -| Git diff | Noisy (cell outputs embedded) | Clean (no embedded outputs) | -| Local data | Manual file loading | Drop CSV in `data/`, query immediately | - -Both are Git-trackable. DQL notebook is purpose-built for SQL analytics without writing any Python. - -## I just want to explore a CSV file. Can DQL help? - -Yes. This is the fastest DQL workflow: - -```bash -npm install -g @duckcodeailabs/dql-cli -dql init myproject && cd myproject -dql notebook -``` - -Drop your CSV into the `data/` folder. In the notebook, query it immediately: - -```sql -SELECT * FROM read_csv_auto('data/yourfile.csv') LIMIT 20 -``` - -No warehouse, no credentials, no configuration. DuckDB reads the file directly. - -## Can I preview visualizations locally? - -Yes. The recommended local flow is: - -```bash -dql init my-dql-project -cd my-dql-project -dql new block "Pipeline Health" -dql preview blocks/pipeline_health.dql --open -``` - -This uses local sample data and the built-in preview server. - -## Do I need a cloud warehouse to try DQL? - -No. - -The easiest first-run experience uses local CSV or Parquet data with the `file` or `duckdb` connector path. That is the recommended open-source evaluation flow. - -## What is the difference between `custom` and `semantic` blocks? - -- `type = "custom"` means the block executes SQL declared in the block itself. -- `type = "semantic"` means the block references a semantic-layer metric and should not contain its own SQL query. - -If you are starting fresh, use `custom` blocks first. - -## What syntax should I start with? - -For open-source adoption, start with the reusable block syntax shown in the starter template. - -DQL also supports dashboard and chart-call syntax, but the easiest first path is: - -- `block { ... }` -- local data -- `dql preview` - -## Can DQL be used from code, not just the CLI? - -Yes. - -The repo provides reusable packages for parsing, compiling, rendering, registry management, connectors, governance, and editor integration. - -Good entry points: - -- `@duckcodeailabs/dql-core` -- `@duckcodeailabs/dql-compiler` -- `@duckcodeailabs/dql-runtime` -- `@duckcodeailabs/dql-connectors` - -## What commands are most important for a new user? - -Start with these: - -- `dql init` -- `dql doctor` -- `dql notebook` -- `dql parse` -- `dql preview` -- `dql build` -- `dql serve` - -These cover setup, validation, interactive exploration, and sharing. - -## Does `dql test` execute assertions against a real database? - -It can, but only when a working execution path is configured. - -Without a runnable database connection, test flows are limited to structural discovery or dry-run behavior. For first-time local exploration, focus on `parse`, `preview`, `build`, and `serve` first. - -## What is not included in the open DQL repo? - -This repo does not include: - -- Natural-language or agentic block generation -- MCP runtime -- Approval workflows and product orchestration - -Those are separate from the standalone open-source DQL language/tooling layer. The notebook UI (`dql notebook`) is fully included. - -## How is DQL different from dbt? - -dbt is primarily a transformation workflow and semantic modeling system. - -DQL is focused on durable analytics answer assets: blocks that package query logic, parameters, visualization, and tests into reusable artifacts. - -Many teams can use both together rather than choosing one over the other. - -## How do I get started fastest? - -Fastest DQL-only path: - -```bash -npm install -g @duckcodeailabs/dql-cli -dql init my-dql-project -cd my-dql-project -dql doctor -dql notebook -``` - -If you want the dbt-integrated path, start from `dbt-labs/Semantic-Layer-Online-Course` so DQL can pick up dbt semantic models and metrics immediately. - -Then read: - -- [Getting Started](./getting-started.md) -- [Examples](./examples.md) -- [Project Config](./project-config.md) -- [Data Sources](./data-sources.md) diff --git a/docs/migration-guides/README.md b/docs/migration-guides/README.md deleted file mode 100644 index 4f7ca75a..00000000 --- a/docs/migration-guides/README.md +++ /dev/null @@ -1,29 +0,0 @@ -# Migration Guides - -These guides help you move existing analytics assets into DQL using the current open-source workflow. - -The OSS migration path today is intentionally practical and manual-first: - -- use your existing SQL, metrics, or saved queries as source material -- scaffold a DQL block or semantic block -- validate it with `dql parse` -- preview it locally when applicable -- add tests and metadata before wider reuse - -## Guides - -- [Raw SQL to DQL](./raw-sql.md) -- [dbt Metric to Semantic Block](./dbt.md) -- [Saved BI Query to DQL Block](./saved-bi-query.md) - -## Related CLI Command - -You can also use: - -```bash -dql migrate raw-sql -dql migrate dbt --input ./my-dbt-project -dql migrate metabase -``` - -In the open-source CLI, `dql migrate` is scaffold-only. It provides a starting point, not a full automatic migration. diff --git a/docs/migration-guides/dbt.md b/docs/migration-guides/dbt.md deleted file mode 100644 index f8debf1d..00000000 --- a/docs/migration-guides/dbt.md +++ /dev/null @@ -1,88 +0,0 @@ -# dbt Metric to Semantic Block - -This guide shows how to represent a dbt metric or semantic concept as a DQL semantic block. - -## When to use this guide - -Use this when you already have: - -- a dbt metric -- semantic-layer YAML -- a governed business metric you do not want to re-encode as raw SQL in every block - -## Goal - -Model the analytics asset as a DQL semantic block that references a metric rather than embedding SQL directly. - -## Step 1: Start with the metric name - -Suppose the source metric is: - -- `annual_recurring_revenue` - -## Step 2: Create a semantic block - -Create `blocks/arr_growth.dql`: - -```dql -block "ARR Growth" { - domain = "finance" - type = "semantic" - description = "Semantic block for annual recurring revenue" - owner = "finance-analytics" - tags = ["finance", "arr", "semantic"] - metric = "annual_recurring_revenue" -} -``` - -## Step 3: Add semantic-layer metadata - -Create a metric definition in `semantic-layer/metrics/annual_recurring_revenue.yaml`: - -```yaml -name: annual_recurring_revenue -label: Annual Recurring Revenue -description: Finance-approved recurring revenue metric -domain: finance -sql: SUM(arr_amount) -type: sum -table: finance_arr -owner: finance-analytics -``` - -## Step 4: Validate the block - -```bash -dql parse blocks/arr_growth.dql -dql info blocks/arr_growth.dql -``` - -## Important rule - -For a semantic block: - -- keep `type = "semantic"` -- include `metric = "..."` -- do not start by adding a raw SQL `query` field - -If your goal is to run local preview quickly, start with a `custom` block first. Semantic blocks are best when your metric contract already exists and you want DQL to reference that contract. - -## When not to use a semantic block - -Do not use `type = "semantic"` just because a query is important. - -Use `type = "custom"` if: - -- you want to author the SQL in the block itself -- you want local CSV or DuckDB preview immediately -- you do not yet have a stable metric definition - -## CLI helper - -You can inspect the scaffold flow with: - -```bash -dql migrate dbt --input ./my-dbt-project -``` - -In the OSS CLI this is still scaffold-only, so treat it as planning support, not full automatic conversion. diff --git a/docs/migration-guides/raw-sql.md b/docs/migration-guides/raw-sql.md deleted file mode 100644 index 8e0be8ec..00000000 --- a/docs/migration-guides/raw-sql.md +++ /dev/null @@ -1,136 +0,0 @@ -# Raw SQL to DQL - -This guide shows how to turn an existing SQL query into a reusable DQL block. - -## When to use this guide - -Use this when you already have: - -- an ad hoc SQL query -- a saved SQL file -- a notebook query you want to keep - -## Goal - -Take a SQL query and wrap it in a DQL block with: - -- metadata -- optional parameters -- optional visualization -- basic assertions - -## Starting SQL - -```sql -SELECT - segment, - SUM(amount) AS revenue -FROM fct_revenue -GROUP BY segment -ORDER BY revenue DESC; -``` - -## Step 1: Create a DQL block - -Create `blocks/revenue_by_segment.dql`: - -```dql -block "Revenue by Segment" { - domain = "revenue" - type = "custom" - description = "Revenue grouped by customer segment" - owner = "finance-analytics" - tags = ["revenue", "segment", "migration"] - - query = """ - SELECT - segment, - SUM(amount) AS revenue - FROM fct_revenue - GROUP BY segment - ORDER BY revenue DESC - """ - - visualization { - chart = "bar" - x = segment - y = revenue - } - - tests { - assert row_count > 0 - } -} -``` - -## Step 2: Validate the block - -```bash -dql parse blocks/revenue_by_segment.dql -``` - -## Step 3: Preview it locally - -If your query runs against local files or starter data: - -```bash -dql preview blocks/revenue_by_segment.dql --open -``` - -## Step 4: Add parameters if needed - -If your original query had date filters, region filters, or environment-specific values, move them into `params`. - -Example: - -```dql -params { - period = "current_quarter" -} -``` - -Then reference them in SQL: - -```sql -WHERE fiscal_period = ${period} -``` - -## Step 5: Add a minimal test - -Start with a very small assertion: - -```dql -tests { - assert row_count > 0 -} -``` - -Then expand later with business-specific assertions. - -## Good migration candidates - -Raw SQL is a strong fit for DQL when the query is: - -- reused repeatedly -- shown to business users -- charted in the same way often -- important enough to test and version in Git - -## CLI helper - -You can generate a starting scaffold with: - -```bash -dql migrate raw-sql -``` - -In the OSS CLI this is scaffold-only, so you still need to paste in the real query and review the block manually. - -## Recommended next step - -After migration, run: - -```bash -dql parse blocks/revenue_by_segment.dql -dql build blocks/revenue_by_segment.dql -``` diff --git a/docs/migration-guides/saved-bi-query.md b/docs/migration-guides/saved-bi-query.md deleted file mode 100644 index 9d521512..00000000 --- a/docs/migration-guides/saved-bi-query.md +++ /dev/null @@ -1,116 +0,0 @@ -# Saved BI Query to DQL Block - -This guide shows how to migrate a saved query or card from BI tools such as Metabase, Tableau, or Looker into a reusable DQL block. - -## When to use this guide - -Use this when you already have: - -- a saved BI query -- a chart that business users revisit often -- a card, tile, worksheet, or explore result worth versioning in Git - -## Goal - -Extract the durable parts of the BI artifact and re-create them in DQL: - -- title -- description -- owner -- tags -- SQL query or metric reference -- chart type -- basic tests - -## Step 1: Capture the source asset - -From the original BI tool, collect: - -- the SQL or generated SQL -- chart type -- filters or parameters -- owner/team -- business description - -## Step 2: Create a DQL block - -Example starting point: - -```dql -block "Top Opportunities" { - domain = "sales" - type = "custom" - description = "Migrated from a saved BI query for pipeline review" - owner = "revops" - tags = ["sales", "pipeline", "migration"] - - query = """ - SELECT - account_name, - stage, - amount - FROM opportunities - ORDER BY amount DESC - LIMIT 20 - """ - - visualization { - chart = "table" - } - - tests { - assert row_count > 0 - } -} -``` - -## Step 3: Convert filters into params - -If the BI artifact used filters, move them into DQL params instead of hard-coding them. - -For example: - -```dql -params { - region = "North America" -} -``` - -Then use that in the SQL query. - -## Step 4: Validate and preview - -```bash -dql parse blocks/top_opportunities.dql -dql preview blocks/top_opportunities.dql --open -``` - -## How to choose chart types - -Common mappings: - -- BI table card → `chart = "table"` -- bar chart → `chart = "bar"` -- trend chart → `chart = "line"` -- headline metric → `chart = "kpi"` - -## CLI helper - -You can generate scaffolds with: - -```bash -dql migrate metabase -dql migrate looker -dql migrate tableau -``` - -In the OSS CLI these flows are currently scaffold-only, so plan to review and edit the generated block manually. - -## Good target assets - -Saved BI queries are strong DQL migration candidates when they are: - -- repeatedly used in weekly reviews -- copied across dashboards or teams -- business-critical enough to test -- stable enough to deserve Git history and ownership diff --git a/docs/project-config.md b/docs/project-config.md deleted file mode 100644 index e9ebeb56..00000000 --- a/docs/project-config.md +++ /dev/null @@ -1,127 +0,0 @@ -# Project Config - -`dql.config.json` is the lightweight project-level configuration file used by -the OSS CLI for local preview and bundle serving. - -## Location - -Place `dql.config.json` at the root of your DQL project. - -Example layout: - -```text -my-dql-project/ - blocks/ - data/ - semantic-layer/ - dql.config.json -``` - -The CLI walks upward from the current file or directory until it finds this -config file. - ---- - -## Example - -```json -{ - "project": "my-dql-project", - "defaultConnection": { - "driver": "file", - "filepath": ":memory:" - }, - "dataDir": "./data", - "preview": { - "port": 3474, - "open": true, - "theme": "light" - } -} -``` - ---- - -## Fields - -### `project` - -Human-readable project name used in starter scaffolds and future tooling. - -### `defaultConnection` - -Connection settings used by `dql preview` and `dql serve`. - -Supported Phase 1 local-first choices: - -- `file` — recommended for local experimentation with `read_csv_auto(...)`, `read_parquet(...)`, and other DuckDB file readers -- `duckdb` — use a DuckDB database file or `:memory:` -- `sqlite`, `postgresql`, `mysql`, `snowflake`, `bigquery`, `mssql` — available through connectors, but not required for first-run adoption - -Example local file connection: - -```json -{ - "defaultConnection": { - "driver": "file", - "filepath": ":memory:" - } -} -``` - -Example DuckDB database file: - -```json -{ - "defaultConnection": { - "driver": "duckdb", - "filepath": "./local/dev.duckdb" - } -} -``` - -For more connector-specific examples, see [Data Sources](./data-sources.md). - -### `dataDir` - -Convenience convention for local project data. The CLI does not rewrite queries -to use this path; it is there so teams have one obvious place for sample data. - -### `preview` - -Controls local browser workflows. - -- `port` — preferred port for `preview` and `serve` -- `open` — whether to open a browser automatically -- `theme` — compiler theme passed to local preview/build flows - ---- - -## Commands That Use It - -- `dql doctor` -- `dql preview` -- `dql build` -- `dql serve` - ---- - -## Recommended Phase 1 Setup - -For easiest adoption, start with: - -```json -{ - "defaultConnection": { - "driver": "file", - "filepath": ":memory:" - }, - "preview": { - "open": true, - "port": 3474, - "theme": "light" - } -} -``` - -This works well with starter queries that read CSV or Parquet directly. diff --git a/docs/project-structure.md b/docs/project-structure.md deleted file mode 100644 index 99a1f1cf..00000000 --- a/docs/project-structure.md +++ /dev/null @@ -1,61 +0,0 @@ -# Project Structure - -The public DQL workflow is Git-native. A typical project looks like this: - -```text -my-dql-project/ - dql.config.json # Project config: connection, semantic layer, port - blocks/ - clean_orders.dql # Data domain block - revenue_by_segment.dql # Finance domain block (uses ref("clean_orders")) - exec_dashboard.dql # Executive domain block (uses ref("revenue_by_segment")) - semantic-layer/ - metrics/ - revenue.yaml # Metric definitions (appear in lineage) - dimensions/ - segment.yaml # Dimension definitions - hierarchies/ - revenue_time.yaml # Drill paths - cubes/ - revenue_cube.yaml # Cube definitions - data/ - orders.csv # Local sample data - revenue.csv - notebooks/ - welcome.dqlnb # Interactive notebook - dashboards/ # Optional: dashboard .dql files - workbooks/ # Optional: workbook .dql files -``` - -## Conventions - -- **`blocks/`** holds durable reusable DQL blocks — one block per `.dql` file -- **`ref("block_name")`** in SQL creates explicit dependencies between blocks, tracked in lineage -- **`domain`** field on blocks creates business domain groupings — lineage tracks cross-domain flows -- **`semantic-layer/`** defines metrics, dimensions, hierarchies, and cubes in YAML — these appear as nodes in the lineage graph -- **`notebooks/`** stores interactive `.dqlnb` notebooks for exploration -- **`data/`** holds local CSV/Parquet files for DuckDB-powered local analysis - -## Recommended Workflow - -1. **Explore** — `dql notebook` to prototype queries against your data -2. **Author** — `dql new block "Name"` to create a reusable block; use `ref()` for dependencies -3. **Validate** — `dql parse` for syntax, `dql certify` for governance -4. **Lineage** — `dql lineage` to see data flow, `dql lineage --impact` before changes -5. **Commit** — Git review ensures a second set of eyes on ownership, tests, and dependencies - -## Data Flow Example - -``` -source table: orders - ↓ reads_from -block: clean_orders (domain: data, owner: data-team) - ↓ feeds_into (via ref()) -block: revenue_by_segment (domain: finance, owner: finance-team) - ↓ feeds_into (via ref()) -block: exec_dashboard (domain: executive, owner: ceo) - ↓ visualizes -chart: bar (segment × revenue) -``` - -At every node: who owns it, which domain, and certification status. `dql lineage` shows this automatically. diff --git a/docs/publishing.md b/docs/publishing.md deleted file mode 100644 index ecb6e446..00000000 --- a/docs/publishing.md +++ /dev/null @@ -1,197 +0,0 @@ -# Publishing @duckcodeailabs/dql-* to npm - -This document is for maintainers releasing the open-source DQL packages and VS Code extension. - -If you are trying DQL as a user, start with: - -- [Why DQL](./why-dql.md) -- [Getting Started](./getting-started.md) - -This repository publishes the public DQL packages to npm and the editor extension to the VS Code marketplace. - ---- - -## Prerequisites - -- An npm account that is a member of the `@dql` organisation -- `npm whoami` returns your username (run `npm login` if not) -- pnpm 9+, Node.js 18+ -- GitHub Actions secrets configured on the repository: - - `NPM_TOKEN` — for `@duckcodeailabs/dql-*` package publishing - - `VSCE_PAT` — for VS Code extension publishing from CI (optional) - ---- - -## Package Dependency Graph - -The release script (`scripts/release-packages.mjs`) publishes packages in the following order, which respects the dependency graph: - -``` -@duckcodeailabs/dql-core (no @duckcodeailabs/dql-* deps) -@duckcodeailabs/dql-compiler (depends on @duckcodeailabs/dql-core) -@duckcodeailabs/dql-runtime (no @duckcodeailabs/dql-* deps) -@duckcodeailabs/dql-charts (no @duckcodeailabs/dql-* deps; peer deps: react) -@duckcodeailabs/dql-project (depends on @duckcodeailabs/dql-core) -@duckcodeailabs/dql-governance (depends on @duckcodeailabs/dql-core, @duckcodeailabs/dql-project) -@duckcodeailabs/dql-connectors (no @duckcodeailabs/dql-* deps) -@duckcodeailabs/dql-notebook (depends on @duckcodeailabs/dql-core, @duckcodeailabs/dql-connectors) -@duckcodeailabs/dql-lsp (depends on @duckcodeailabs/dql-core) -@duckcodeailabs/dql-cli (depends on @duckcodeailabs/dql-core, @duckcodeailabs/dql-compiler, - @duckcodeailabs/dql-project, @duckcodeailabs/dql-governance, - @duckcodeailabs/dql-connectors, @duckcodeailabs/dql-notebook) -``` - ---- - -## Bumping Versions - -All packages share a version number. Update every `package.json` together before publishing. - -```bash -# Patch release (e.g. 0.1.0 → 0.1.1): -pnpm --recursive exec npm version patch --no-git-tag-version - -# Minor release (e.g. 0.1.0 → 0.2.0): -pnpm --recursive exec npm version minor --no-git-tag-version - -# Then commit the version changes: -git add packages/*/package.json apps/cli/package.json apps/vscode-extension/package.json -git commit -m "chore: bump version to 0.6.1" -``` - ---- - -## Release Workflow - -### Step 1 — Ensure the build and tests are green - -```bash -pnpm build -pnpm test -``` - -### Step 2 — Dry run (inspect tarballs before publishing) - -```bash -pnpm release:dry-run -``` - -This runs `pnpm build && pnpm test` and then packs each package into `.release-artifacts/` using `pnpm pack`. Review the generated `.tgz` files to verify that only the `dist/` directory and `package.json` are included (the `files` field in each `package.json` is set to `["dist"]`). - -### Step 3 — Tag the release - -```bash -git tag -a v0.7.0 -m "DQL v0.7.0" -git push origin v0.7.0 -``` - -### Step 4 — Publish - -**Via CI (recommended):** Pushing the tag triggers the GitHub `Release` workflow, which publishes all packages automatically when `NPM_TOKEN` is set. - -**Manually (if CI is unavailable):** - -```bash -pnpm release:publish -``` - -This runs `scripts/release-packages.mjs --publish`, which calls `pnpm publish --access public --no-git-checks` for each package in dependency order: - -```bash -# The script iterates through: -packages/dql-core -packages/dql-compiler -packages/dql-runtime -packages/dql-charts -packages/dql-project -packages/dql-governance -packages/dql-connectors -packages/dql-notebook -packages/dql-lsp -apps/cli -``` - -**To publish a single package manually:** - -```bash -cd packages/dql-core -pnpm publish --access public -``` - -`--access public` is required because all packages are scoped (`@duckcodeailabs/dql-`) and scoped packages default to private on npm. Each `package.json` also carries `"publishConfig": { "access": "public" }` as a safety net. - ---- - -## Publishing the VS Code Extension - -The VS Code extension (`apps/vscode-extension`) is published to the VS Code Marketplace using `vsce`, not npm. - -```bash -# Package into a .vsix file: -pnpm release:extension -# Equivalent to: -cd apps/vscode-extension -pnpm run package # produces dql-language-support-.vsix - -# Publish to the marketplace (requires VSCE_PAT): -cd apps/vscode-extension -VSCE_PAT= pnpm run publish:vsce -``` - -Obtain a Personal Access Token from https://dev.azure.com under the `dql` publisher account. - ---- - -## GitHub Release - -After all packages are published, create a GitHub release against the tag: - -```bash -gh release create v0.7.0 \ - --title "v0.7.0 — Project Manifest & Enhanced Lineage" \ - --notes "## Highlights - -- **\`dql compile\`** — generates \`dql-manifest.json\` project artifact (like dbt's manifest.json) -- **dbt manifest import** — \`dql compile --dbt-manifest\` connects dbt lineage as upstream -- **Notebook lineage** — \`.dqlnb\` SQL/DQL cells included in lineage and manifest -- **Smart lineage lookup** — \`dql lineage orders\` auto-resolves to table, block, or metric -- **\`--table\` / \`--metric\` flags** — explicit type lookup for lineage queries -- **Impact analysis on any node** — \`dql lineage --impact orders\` works on tables, metrics, blocks -- **DuckDB reader extraction** — \`read_csv_auto()\` / \`read_parquet()\` tracked as source tables -- **Rich lineage summary** — data flow DAG tree, block details, cross-domain flows - -Published packages: -- [@duckcodeailabs/dql-cli@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-cli) -- [@duckcodeailabs/dql-core@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-core) -- [@duckcodeailabs/dql-compiler@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-compiler) -- [@duckcodeailabs/dql-connectors@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-connectors) -- [@duckcodeailabs/dql-runtime@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-runtime) -- [@duckcodeailabs/dql-charts@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-charts) -- [@duckcodeailabs/dql-project@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-project) -- [@duckcodeailabs/dql-governance@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-governance) -- [@duckcodeailabs/dql-notebook@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-notebook) -- [@duckcodeailabs/dql-lsp@0.7.0](https://www.npmjs.com/package/@duckcodeailabs/dql-lsp)" -``` - ---- - -## Notes - -- Do not publish packages from the closed DuckCode product from this repository. -- `@duckcodeailabs/dql-cli` registers a `dql` binary on install. -- `dql-language-support` is packaged separately through `vsce`, not `npm publish`. -- `v0.8.7` is the current release. All packages use unified versioning — bump all together. - ---- - -## First Release Checklist - -- [ ] `pnpm build` passes for all packages -- [ ] `pnpm test` passes for all packages -- [ ] Version bumped consistently in all `package.json` files -- [ ] `pnpm release:dry-run` completed; tarballs reviewed -- [ ] `NPM_TOKEN` secret set in GitHub repository settings -- [ ] `VSCE_PAT` secret set (if publishing extension from CI) -- [ ] Release tag created and pushed: `git push origin v` -- [ ] GitHub release created via `gh release create` -- [ ] VS Code extension published (if applicable) diff --git a/docs/repo-testing.md b/docs/repo-testing.md deleted file mode 100644 index 0ce35171..00000000 --- a/docs/repo-testing.md +++ /dev/null @@ -1,115 +0,0 @@ -# Repo Testing - -This guide shows how to validate the full open-source DQL repo from a real source checkout. - -## When to use this guide - -Use this if you want to: - -- verify the monorepo builds cleanly -- smoke-test the CLI from source -- test the browser notebook end-to-end -- validate the getting-started flow before release - -## Prerequisites - -- Node.js 18, 20, or 22 LTS -- `pnpm` 9+ -- a fresh clone of the DQL repo - -## 1. Install and build from source - -```bash -git clone https://github.com/duckcode-ai/dql.git -cd dql -pnpm install -pnpm build -pnpm test -``` - -Sanity-check the CLI entrypoint: - -```bash -pnpm --filter @duckcodeailabs/dql-cli exec dql --help -``` - -The rest of this guide assumes you stay at the repo root and invoke the CLI the same way. - -## 2. Smoke-test init and notebook - -The fastest confidence pass is to scaffold a project, run the local checks, and open the notebook. - -### Basic init (no dbt) - -```bash -rm -rf /tmp/dql-smoke -pnpm --filter @duckcodeailabs/dql-cli exec dql init /tmp/dql-smoke -pnpm --filter @duckcodeailabs/dql-cli exec dql doctor /tmp/dql-smoke -pnpm --filter @duckcodeailabs/dql-cli exec dql notebook /tmp/dql-smoke -``` - -### With Jaffle Shop dbt project - -```bash -git clone https://github.com/dbt-labs/Semantic-Layer-Online-Course.git /tmp/jaffle-shop -cd /tmp/jaffle-shop -pip install dbt-duckdb && dbt deps && dbt build --profiles-dir . -pnpm --filter @duckcodeailabs/dql-cli exec dql init . -pnpm --filter @duckcodeailabs/dql-cli exec dql doctor . -pnpm --filter @duckcodeailabs/dql-cli exec dql notebook . -``` - -Use the semantic-layer course repo here so the smoke test covers the dbt semantic models and metrics that DQL is expected to import. - -## 3. Validate block workflows - -From a scaffolded project: - -```bash -cd /tmp/dql-smoke -pnpm --filter @duckcodeailabs/dql-cli exec dql new block "Test Block" --domain test -pnpm --filter @duckcodeailabs/dql-cli exec dql parse blocks/test_block.dql -pnpm --filter @duckcodeailabs/dql-cli exec dql certify blocks/test_block.dql -``` - -## 4. Manual browser checklist - -After opening the notebook, verify these flows manually: - -- the notebook opens at `http://127.0.0.1:` -- the welcome notebook loads automatically -- the file sidebar lists project assets -- a DQL cell runs and returns rows -- a SQL cell runs and returns rows -- a chart cell can link to a DQL or SQL cell -- the connection panel renders the available drivers -- the local `file` connection can be saved and tested -- notebook export downloads a `.dqlnb` file - -## 5. Preview and serve compatibility - -The new notebook flow should not break the classic preview flow. - -Run this from a scaffolded project with a block: - -```bash -pnpm --filter @duckcodeailabs/dql-cli exec dql preview /tmp/dql-smoke/blocks/test_block.dql --open -pnpm --filter @duckcodeailabs/dql-cli exec dql build /tmp/dql-smoke/blocks/test_block.dql -``` - -## 6. Release confidence checklist - -Before publishing or tagging a release, confirm: - -- `pnpm build` passes from the repo root -- `pnpm test` passes from the repo root -- `dql --help` shows `notebook` and all commands -- `dql init` scaffolds correctly (with and without dbt project) -- the notebook opens and queries run -- at least one block builds successfully - -## Troubleshooting - -- If DuckDB bindings fail after changing Node versions, rerun `pnpm install`. -- If port `3474` is busy, pass `--port 4474` to `dql notebook`, `dql preview`, or `dql serve`. -- If `dql` is not on your shell `PATH`, run it with `pnpm --filter @duckcodeailabs/dql-cli exec dql` from the repo root. diff --git a/docs/v1-support-scope.md b/docs/v1-support-scope.md deleted file mode 100644 index a6c3ea4e..00000000 --- a/docs/v1-support-scope.md +++ /dev/null @@ -1,20 +0,0 @@ -# V1 Support Scope - -This repository supports the open DQL adoption layer. - -## Supported - -- DQL syntax and parsing -- formatting and semantic analysis -- block testing and certification primitives -- project and registry primitives -- VS Code language support -- examples and starter templates - -## Not supported here - -- DuckCode notebook coworker -- natural-language generation -- source approvals -- MCP runtime -- product APIs and desktop/container packaging diff --git a/docs/examples/jaffle-shop-lineage-demo.dqlnb b/examples/gallery/jaffle-shop-lineage-demo.dqlnb similarity index 100% rename from docs/examples/jaffle-shop-lineage-demo.dqlnb rename to examples/gallery/jaffle-shop-lineage-demo.dqlnb