From 7cb7b071e28a50fd51d80dd473f7c6adb4c2c05e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Benjamin=20Egelund-M=C3=BCller?= Date: Thu, 8 Jan 2026 16:27:01 +0100 Subject: [PATCH] Docs for how reconciles work internally --- docs/docs/internals/_category_.yml | 4 ++ docs/docs/internals/local-vs-cloud.md | 49 ++++++++++++++++++++++++ docs/docs/internals/project-execution.md | 44 +++++++++++++++++++++ 3 files changed, 97 insertions(+) create mode 100644 docs/docs/internals/_category_.yml create mode 100644 docs/docs/internals/local-vs-cloud.md create mode 100644 docs/docs/internals/project-execution.md diff --git a/docs/docs/internals/_category_.yml b/docs/docs/internals/_category_.yml new file mode 100644 index 00000000000..5f6dea6cb05 --- /dev/null +++ b/docs/docs/internals/_category_.yml @@ -0,0 +1,4 @@ +position: 80 +label: Internals +collapsible: true +collapsed: true diff --git a/docs/docs/internals/local-vs-cloud.md b/docs/docs/internals/local-vs-cloud.md new file mode 100644 index 00000000000..45c932a88dd --- /dev/null +++ b/docs/docs/internals/local-vs-cloud.md @@ -0,0 +1,49 @@ + +--- +title: Local vs. Cloud +description: Explaining Rill's local and cloud environments +sidebar_label: Local vs. Cloud +sidebar_position: 20 +--- + +# Local vs. Cloud + +Rill has two complementary environments: + +## Rill Developer (Local) + +The local CLI application (`rill`) for building projects. + +The most notable command is `rill start `, which starts a long-running process that: +- Watches files and re-executes on change +- Serves an IDE at `http://localhost:9009` +- Auto-syncs environment variables with Rill Cloud (if authenticated) + +Additionally, the `rill` CLI can authenticate with Rill Cloud, storing an access token in `~/.rill`. This unlocks a variety of management commands for Rill Cloud, including: +- Managing orgs, projects, users, usergroups and service accounts +- Changing roles and access restrictions +- Viewing project status, logs, and resource state +- Triggering project refreshes, including granular refresh of specific resources or model partitions + +## Rill Cloud + +The managed cloud service for production deployments. Usually accessed on `ui.rilldata.com` or `api.rilldata.com`. + +**Deployment options:** +- Connect a GitHub repository for continuous deploys on push +- Manual deploy from CLI or local IDE + +**Production features:** +- User management and RBAC +- Data orchestration at scale +- Dashboard serving and monitoring +- UI for creating alerts and reports +- Link-based sharing and embedding + +## Integration Points + +The local and cloud environments connect at several points: +- **Authentication**: Local CLI authenticates with Cloud via OAuth +- **Environment variables**: `rill start` automatically syncs env vars with the connected Cloud project +- **Project identification**: Cloud project is identified by the Git remote in the local directory +- **Cloud operations**: Local CLI can manage Cloud projects and user access (`rill -h` for details) diff --git a/docs/docs/internals/project-execution.md b/docs/docs/internals/project-execution.md new file mode 100644 index 00000000000..19e223de496 --- /dev/null +++ b/docs/docs/internals/project-execution.md @@ -0,0 +1,44 @@ +--- +title: Project execution +description: How Rill parses and executes project resources +sidebar_label: Project execution +sidebar_position: 10 +--- + +# Project Execution + +When you save a file in a Rill project, it flows through a pipeline of components: + +``` + Files → Watcher → Parser → Catalog → Controller → Reconciler + ↓ ↓ ↓ + Parse errors DAG order Reconcile errors +``` + +1. **Watcher**: Detects file changes in the project directory. +2. **Parser**: Converts files into internal resource definitions and organizes them into a dependency graph (DAG). +3. **Catalog**: Persistent store holding the declared "spec" and current "state" of each resource. +4. **Controller**: Watches the catalog and triggers reconciliation for changed resources in DAG order. +5. **Reconciler**: Executes actions to make each resource's current state match its declared spec. + +## Resource Lifecycle + +Each resource type has its own reconciler containing domain-specific logic. For example, the `model` reconciler runs SQL queries and tracks ingestion state, while the `metrics_view` reconciler validates dimensions and measures. + +Some reconcilers are always cheap; others can trigger slow or costly operations. The `model` reconciler is notably expensive when it: +- Re-ingests data from an external database +- Re-creates tables with complex SQL transformations + +Reconcilers are designed to run fast when no action is needed—if a model already exists, its reconciler returns quickly. This allows Rill to trigger reconciliation liberally. Reconcilers run when: +- A parent resource in the DAG finishes reconciling +- The runtime restarts +- `rill.yaml` or environment variables change +- A scheduled time is reached (to honor `cron` refresh expressions) + +## Internal Resources + +Some resources exist without corresponding files: +- `project_parser`: Global resource that handles file parsing and stores parse errors. +- `alert` and `report`: Can be created via the Rill Cloud UI instead of files. +- `component`: Created internally as part of a `canvas` resource. +- `explore`: Can optionally be created as part of a `metrics_view` resource.