From a477172e22e433d90299805cb323647ba5997271 Mon Sep 17 00:00:00 2001 From: Maximum Code Date: Fri, 24 Apr 2026 19:52:43 -0700 Subject: [PATCH 01/10] Add docs reference pages and improve SQL autocomplete --- RELEASE_NOTES.md | 189 ++- docs/architecture.md | 6 +- docs/cli.md | 258 --- docs/getting-started.md | 8 +- docs/internals.md | 396 ----- docs/mcp-server.md | 235 --- docs/releases/v3.4.0-pr-notes.md | 82 + docs/rest-api.md | 635 -------- docs/roadmap.md | 15 +- docs/storage-inspector.md | 202 --- samples/README.md | 4 +- .../Helpers/SqlCompletionProvider.cs | 651 ++++++++ .../Shared/SqlCompletionProviderTests.cs | 244 +++ www/architecture-reference.html | 1140 ++++++++++++++ www/architecture.html | 3 + .../csharpdb-vs-sqlite-benchmark-guide.html | 3 + ...rpdb-vs-sqlite-benchmarking-reference.html | 516 ++++++ www/css/style.css | 71 + www/docs/getting-started-reference.html | 688 ++++++++ www/docs/getting-started.html | 3 + www/docs/internals.html | 3 +- www/docs/performance-reference.html | 662 ++++++++ www/docs/performance.html | 3 + www/docs/pipelines.html | 3 + www/docs/query-execution-pipeline.html | 679 ++++++++ www/docs/sql-reference.html | 601 +++++++ www/docs/sql.html | 3 + www/docs/storage-engine-reference.html | 1388 +++++++++++++++++ www/docs/storage-engine.html | 3 + www/roadmap-reference.html | 558 +++++++ www/roadmap.html | 3 + www/sitemap.xml | 48 + 32 files changed, 7481 insertions(+), 1822 deletions(-) delete mode 100644 docs/cli.md delete mode 100644 docs/internals.md delete mode 100644 docs/mcp-server.md create mode 100644 docs/releases/v3.4.0-pr-notes.md delete mode 100644 docs/rest-api.md delete mode 100644 docs/storage-inspector.md create mode 100644 src/CSharpDB.Admin/Helpers/SqlCompletionProvider.cs create mode 100644 tests/CSharpDB.Admin.Forms.Tests/Components/Shared/SqlCompletionProviderTests.cs create mode 100644 www/architecture-reference.html create mode 100644 www/blog/csharpdb-vs-sqlite-benchmarking-reference.html create mode 100644 www/docs/getting-started-reference.html create mode 100644 www/docs/performance-reference.html create mode 100644 www/docs/query-execution-pipeline.html create mode 100644 www/docs/sql-reference.html create mode 100644 www/docs/storage-engine-reference.html create mode 100644 www/roadmap-reference.html diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 315890f8..66e401f1 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,94 +1,127 @@ # What's New -## v3.3.0 +## v3.4.0 -This release starts from `bf954f788cc4a4c5915af2a9a68b66bc7c06f126` -(`Checkpoint hot-right-edge recovery and add plans 4/5`) and is current -through `8d11653c81e690849f462cb376872d901eabfeb6` -(`Refresh benchmark release docs and guardrails`). +v3.4.0 focuses on daemon packaging, cross-platform deployment, and remote host +consolidation without changing SQL, storage, WAL, query planning, or the gRPC +client contract. -v3.3.0 focuses on durable write performance, storage tuning for embedded -ADO.NET and EF Core usage, same-runner SQLite comparisons, and a cleaner -release benchmark process. The main benchmark story is now promoted from the -April 21, 2026 release-core run with guardrails passing at -`PASS=185, WARN=0, SKIP=0, FAIL=0`. +### Remote Host Consolidation -### Durable Write and Indexing Performance +- `CSharpDB.Daemon` now hosts both the existing REST `/api` surface and gRPC + from one long-running process. +- REST and gRPC requests share the same warm daemon-hosted database client, so + remote users no longer need separate REST and gRPC host processes for the + same database. +- Plain `http://` daemon endpoints support `Transport = Grpc` through + gRPC-Web compatibility so gRPC and REST can share the default service URL. + HTTPS and custom test clients can continue using native gRPC. +- REST is enabled in the daemon by default and can be disabled with + `CSharpDB__Daemon__EnableRestApi=false`. +- The standalone `CSharpDB.Api` REST host remains supported for REST-only + deployments. +- Release archive smoke validation now calls the daemon REST `/api/info` + endpoint and a gRPC `GetInfoAsync` client after extracting and starting each + daemon binary. -- Added append-optimized index storage paths for row-id chains and hashed - payloads, including appendable payload codecs, overflow-store improvements, - and focused tests for the new insert-maintenance paths. -- Optimized indexed insert maintenance for monotonic and append-heavy - workloads, including hot right-edge recovery, insert sequence context, and - expanded B-tree/index diagnostics for commit-path investigation. -- Added trailing-integer support for composite grouped aggregate planning while - tightening SQL index metadata defaults so multi-column indexes no longer - receive trailing-integer hash options unless explicitly requested. -- Expanded record encoding and serialization support used by the optimized - storage paths and added compatibility coverage for hashed index payloads, - append-only row-id chains, record encoding, collation metadata, and SQL index - behavior. +### Admin Warm Local Database Hosting -### Embedded ADO.NET and EF Core Storage Tuning +- `CSharpDB.Admin` direct local mode keeps a warm in-process database instance + and now opens it through hybrid incremental-durable database options by + default. +- Admin startup and database switching both use the same host database option + builder, so opening a different database from the UI keeps the same warm + local database behavior. +- Admin remote mode still uses `CSharpDB:Transport` plus `CSharpDB:Endpoint` + without attaching local direct/hybrid options. +- Set `CSharpDB__HostDatabase__OpenMode=Direct` to opt back into the older + plain direct open path for local Admin runs. +- Admin table data filters now support contains, starts-with, and ends-with + `LIKE` placement plus exact `=` match mode per column. +- The Admin SQL query editor now has homegrown guided completions for SQL + keywords, table/view selection, select-list columns, qualified alias columns, + and stored procedure names without adding a third-party editor dependency. -- Added storage tuning presets for embedded ADO.NET and EF Core users, - including `CSharpDbStoragePreset`, embedded open modes, connection-string - builder support, and configuration resolution for shared file-backed usage. -- Updated the EF Core provider option validation and relational connection - setup so storage mode and connection behavior are explicit for embedded - workloads. -- Added comparative ADO.NET and EF Core smoke coverage plus embedded storage - tuning tests for the new configuration surface. -- Updated package references across the CLI, EF Core sample/provider, tests, - and benchmark projects. +### Daemon Service Packaging -### Benchmarks and SQLite Comparisons +- Added Windows Service and systemd host integration to `CSharpDB.Daemon`. + These hooks are no-ops for normal console and `dotnet run` execution. +- Added Windows service install/uninstall scripts with defaults for + `CSharpDBDaemon`, `C:\Program Files\CSharpDB\Daemon`, + `C:\ProgramData\CSharpDB`, and `http://127.0.0.1:5820`. +- Added Linux systemd service template plus install/uninstall scripts with + defaults for `/opt/csharpdb-daemon`, `/var/lib/csharpdb`, service user + `csharpdb`, and `http://127.0.0.1:5820`. +- Added macOS launchd plist template plus install/uninstall scripts with + defaults for `/usr/local/lib/csharpdb-daemon`, + `/usr/local/var/csharpdb`, and `http://127.0.0.1:5820`. -- Added durable SQLite comparison coverage, including SQLite C API helpers, - concurrent SQLite C API benchmarks, concurrent ADO.NET comparison - benchmarks, strict insert comparison rows, and EF Core comparison benchmarks. -- Added CSharpDB-versus-SQLite performance guidance and blog content under - `docs/query-and-durable-write-performance`, with website and sitemap updates - for the new comparison material. -- Replaced the older programmatic insert planning docs with the current - ADO.NET/EF storage tuning guide and release-core benchmark story. +### Release Archives -### Release Benchmark Process +- Added `scripts/Publish-CSharpDbDaemonRelease.ps1` for self-contained, + single-file, non-trimmed daemon release archives. +- Added release archive coverage for `win-x64`, `linux-x64`, and `osx-arm64`. +- Added checksum generation through `SHA256SUMS.txt`. +- Updated the GitHub Release workflow to build daemon archives on native + Windows, Linux, and macOS runners, verify each archive, smoke-start the + extracted daemon, and attach the archives plus combined checksums to the + GitHub Release. -- Redesigned `tests/CSharpDB.Benchmarks/README.md` into a compact, - user-facing benchmark contract with a generated core scorecard, current - results, benchmark map, and run/promote instructions. -- Added `BENCHMARK_CATALOG.md`, `HISTORY.md`, `SQLITE_COMPARISON.md`, - `release-core-manifest.json`, and - `scripts/Update-BenchmarkReadme.ps1` so published numbers come from an - explicit manifest and only the generated region is rewritten. -- Added the `--release-core` benchmark command to run the balanced core suite: - master-table, durable SQL batching, concurrent durable writes, hybrid storage - mode, resident hot-set reads, cold open, and SQLite comparison. -- Updated release guardrail comparison logic to support structural `ExtraInfo` - checks and row-specific tolerances for known volatile microbenchmark rows. +### Docs -### Docs, Package READMEs, and Website - -- Refreshed the root README with the current promoted performance numbers: - `1.67M` collection gets/sec, `10.77M` concurrent reader-burst reads/sec, - `798.25K` durable InsertBatch B10000 rows/sec, and `1.04K` concurrent - durable commits/sec. -- Added or refreshed package-local READMEs for Admin, Forms, Reports, CLI, - MCP, Native, API, Data, Engine, EF Core, and the aggregate package surface. -- Updated documentation routes, titles, sitemap entries, and website pages, - including moving public docs under the current `www/docs` structure and - removing unused legacy JavaScript files. +- Updated the daemon README with archive installation, service installation, + upgrade, uninstall, and configuration override guidance. +- Updated the Admin README with warm in-process local database behavior, hybrid + local hosting defaults, and the direct open-mode opt-out. +- Updated the scripts README with daemon packaging and service installer + references. +- Updated the roadmap to mark daemon service packaging done and scoped + cross-platform daemon archive deployment in progress. +- Added a new blog post covering the C# launcher pattern for + `CSharpDB.Admin`, including syntax-highlighted examples for the launcher + executable flow. +- Migrated the remaining source-heavy markdown docs into companion reference + pages under `www` for architecture, getting started, performance, SQL query + execution pipeline, SQL reference, storage engine, roadmap, and the + CSharpDB-versus-SQLite benchmarking article so the full original content now + stays published on the website. +- Updated the curated docs/blog pages and sitemap to point at the new source + reference routes when users need the full original long-form content. +- Removed the duplicated markdown copies of the CLI, REST API, MCP server, + internals, and storage inspector guides after their website versions were + audited and verified. ### Validation -- `dotnet build .\CSharpDB.slnx -c Release --no-restore` completed - successfully. -- `dotnet build .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -c Release --no-restore` - completed successfully during release prep. -- `dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --release-core --repeat 3 --repro` - completed and produced the promoted April 21, 2026 release-core artifacts. -- Final release guardrail comparison passed with - `PASS=185, WARN=0, SKIP=0, FAIL=0`. -- The benchmark README generator was run twice and verified stable on the - second dry run. +- PowerShell parser validation passed for the daemon release publisher and + Windows install/uninstall scripts. +- `bash -n` passed for Linux and macOS service install/uninstall scripts. +- `dotnet restore CSharpDB.slnx` completed successfully. +- `dotnet build CSharpDB.slnx -c Release` completed successfully. +- `dotnet build CSharpDB.slnx -c Release --no-restore` completed successfully. +- `dotnet test tests\CSharpDB.Api.Tests\CSharpDB.Api.Tests.csproj -c Release --no-build` + passed with `15` tests. +- `dotnet test tests\CSharpDB.Daemon.Tests\CSharpDB.Daemon.Tests.csproj -c Release --no-build` + passed with `18` tests. +- `dotnet test tests\CSharpDB.Admin.Forms.Tests\CSharpDB.Admin.Forms.Tests.csproj -c Release --no-build` + passed with `211` tests. +- Content audit confirmed the new source-reference pages preserve the migrated + markdown coverage (100% heading coverage and 99.7-100% token overlap across + the migrated set). +- `python -c "import xml.etree.ElementTree as ET; ET.parse('www/sitemap.xml')"` + passed for the updated site map. +- Repo scan found no remaining references to the deleted duplicated markdown + docs. +- `.\scripts\Publish-CSharpDbDaemonRelease.ps1 -Version 3.4.0 -Runtime win-x64 -OutputRoot artifacts\daemon-release-local` + created `csharpdb-daemon-v3.4.0-win-x64.zip` and `SHA256SUMS.txt`. +- The extracted `win-x64` daemon archive smoke-started successfully, served + `/api/info`, and accepted a gRPC `GetInfoAsync` client call on the same base + URL with a temporary database. +- `dotnet run --project src\CSharpDB.Api\CSharpDB.Api.csproj --configuration Release --no-build --no-launch-profile` + smoke-started successfully and served `/api/info` with a temporary database. +- `dotnet run --project src\CSharpDB.Daemon\CSharpDB.Daemon.csproj --configuration Release --no-build --no-launch-profile` + smoke-started successfully, served `/api/info`, and accepted a gRPC + `GetInfoAsync` client call on the same base URL with a temporary database. +- `dotnet run --project src\CSharpDB.Admin\CSharpDB.Admin.csproj --configuration Release --no-build --no-launch-profile` + smoke-started successfully in direct hybrid incremental-durable mode with a + temporary database. diff --git a/docs/architecture.md b/docs/architecture.md index 1b1c4cc5..981ce3a0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -692,7 +692,7 @@ The current daemon default host shape is: - `UseWriteOptimizedPreset = true` - optional hot-table / hot-collection preload hints through `CSharpDB:HostDatabase` -See the [REST API Reference](rest-api.md) for HTTP details and the [Daemon README](../src/CSharpDB.Daemon/README.md) for the gRPC host design. +See the [REST API Reference](https://csharpdb.com/docs/rest-api.html) for HTTP details and the [Daemon README](../src/CSharpDB.Daemon/README.md) for the gRPC host design. --- @@ -767,7 +767,7 @@ Each row flows upward through the operator chain, transformed at each stage, unt ## See Also - [Getting Started Tutorial](getting-started.md) — Step-by-step walkthrough with code examples -- [Internals & Contributing](internals.md) — How to extend the engine, add SQL statements, create operators -- [REST API Reference](rest-api.md) — HTTP endpoint documentation +- [Internals & Contributing](https://csharpdb.com/docs/internals.html) — How to extend the engine, add SQL statements, create operators +- [REST API Reference](https://csharpdb.com/docs/rest-api.html) — HTTP endpoint documentation - [Roadmap](roadmap.md) — Planned features and project direction - [Benchmark Suite](../tests/CSharpDB.Benchmarks/README.md) — Performance data across all engine layers diff --git a/docs/cli.md b/docs/cli.md deleted file mode 100644 index eb68aba3..00000000 --- a/docs/cli.md +++ /dev/null @@ -1,258 +0,0 @@ -# CSharpDB CLI Reference - -The CSharpDB CLI (`CSharpDB.Cli`) is an interactive REPL for working with CSharpDB databases from the terminal. It supports SQL execution, meta-commands for introspection, transaction management, and batch file execution. - -## Running the CLI - -```bash -# Open or create a database -dotnet run --project src/CSharpDB.Cli -- mydata.db - -# Without arguments, uses csharpdb.db in the current directory -dotnet run --project src/CSharpDB.Cli -``` - -The CLI also supports command mode (non-REPL) for storage diagnostics and maintenance: - -```bash -dotnet run --project src/CSharpDB.Cli -- inspect mydata.db -dotnet run --project src/CSharpDB.Cli -- inspect-page mydata.db 5 --hex -dotnet run --project src/CSharpDB.Cli -- check-wal mydata.db -dotnet run --project src/CSharpDB.Cli -- check-indexes mydata.db --json -dotnet run --project src/CSharpDB.Cli -- migrate-foreign-keys mydata.db --spec fk-spec.json --validate-only -``` - -If the first argument is one of `inspect`, `inspect-page`, `check-wal`, `check-indexes`, or `migrate-foreign-keys`, command mode runs. Otherwise, the CLI opens REPL mode using the first argument as the database path. - -## Interactive Usage - -The REPL displays a `csdb>` prompt. Enter SQL statements followed by a semicolon, or use dot-prefixed meta-commands: - -``` -csdb> CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL, age INTEGER); -OK (0 ms) - -csdb> INSERT INTO users VALUES (1, 'Alice', 30); -1 row(s) affected (1 ms) - -csdb> SELECT * FROM users; -┌────┬───────┬─────┐ -│ id │ name │ age │ -├────┼───────┼─────┤ -│ 1 │ Alice │ 30 │ -└────┴───────┴─────┘ -1 row(s) returned (0 ms) - -csdb> .quit -``` - -Multi-line SQL is supported — the REPL waits for a semicolon before executing. Trigger bodies (`BEGIN ... END;`) are handled correctly. - ---- - -## Meta-Commands - -All meta-commands start with a dot (`.`). They are case-insensitive. - -### Database Introspection - -| Command | Description | -|---------|-------------| -| `.info` | Show database status: table, index, view, trigger, and collection counts | -| `.tables [PATTERN\|--all]` | List tables. By default, collection backing tables are hidden. Use a pattern to filter or `--all` to include internal tables. | -| `.schema [TABLE\|--all]` | Show the CREATE TABLE schema for one or all tables | -| `.indexes [TABLE]` | List indexes, optionally filtered by table name | -| `.views` | List all views | -| `.view ` | Show the CREATE VIEW SQL for a specific view | -| `.triggers [TABLE]` | List triggers, optionally filtered by table name | -| `.trigger ` | Show the CREATE TRIGGER SQL for a specific trigger | -| `.collections` | List document collections (NoSQL API) | - -### Transaction Management - -| Command | Description | -|---------|-------------| -| `.begin` | Begin an explicit transaction | -| `.commit` | Commit the current transaction | -| `.rollback` | Rollback the current transaction | -| `.checkpoint` | Flush WAL pages to the main database file | -| `.backup [--with-manifest]` | Write a committed snapshot backup to a file, optionally with a JSON manifest | -| `.restore [--validate-only]` | Validate a source snapshot or restore it into the current database file; for remote transports the path is resolved on the target host | -| `.migrate-fks [--validate-only] [--backup ]` | Validate or apply foreign-key retrofit migration for existing tables; for remote transports, paths resolve on the connected host | - -### Mode Toggles - -| Command | Description | -|---------|-------------| -| `.timing [on\|off\|status]` | Toggle query timing output (shows execution time in milliseconds) | -| `.snapshot [on\|off\|status]` | Toggle snapshot (read-only) mode for SELECT queries. When enabled, queries use a frozen point-in-time view that is unaffected by concurrent writes. | -| `.syncpoint [on\|off\|status]` | Toggle the sync fast path for primary key lookups. When enabled, cached PK lookups bypass the async pipeline for lower latency. | - -### File Execution - -| Command | Description | -|---------|-------------| -| `.read ` | Execute all SQL statements from a file. Statements are separated by semicolons. Results and errors are printed as they execute. | - -### General - -| Command | Description | -|---------|-------------| -| `.help` | Show a list of all available commands | -| `.quit` / `.exit` | Exit the REPL | - -### Foreign-Key Retrofit Migration - -Use the retrofit workflow when an older database file already contains valid parent/child data but does not yet persist FK metadata: - -```json -{ - "validateOnly": true, - "violationSampleLimit": 100, - "constraints": [ - { - "tableName": "orders", - "columnName": "customer_id", - "referencedTableName": "customers", - "referencedColumnName": "id", - "onDelete": "Restrict" - } - ] -} -``` - -Validate only: - -```bash -dotnet run --project src/CSharpDB.Cli -- migrate-foreign-keys mydata.db --spec fk-spec.json --validate-only -``` - -Apply from the REPL: - -```text -csdb> .migrate-fks fk-spec.json --backup pre-fk.backup.db -``` - -Notes: -- `--validate-only` reports sampled violations without mutating the database. -- `--backup` writes a committed snapshot before apply mode starts. -- For HTTP or gRPC transports, the spec path and backup path are resolved on the connected host. - -## SQL Introspection (`sys.*`) - -You can query metadata with SQL in addition to dot-commands: - -```sql -SELECT * FROM sys.tables ORDER BY table_name; -SELECT * FROM sys.columns WHERE table_name = 'users' ORDER BY ordinal_position; -SELECT * FROM sys.indexes WHERE table_name = 'users'; -SELECT * FROM sys.foreign_keys ORDER BY table_name, column_name; -SELECT * FROM sys.views; -SELECT * FROM sys.triggers; -SELECT * FROM sys.objects ORDER BY object_type, object_name; -SELECT * FROM sys.saved_queries ORDER BY name; -``` - -Underscored aliases are supported: `sys_tables`, `sys_columns`, `sys_indexes`, `sys_foreign_keys`, `sys_views`, `sys_triggers`, `sys_objects`, `sys_saved_queries`. -`sys.columns` includes `is_identity` (0/1) in addition to `is_primary_key`. - ---- - -## Storage Inspector Commands - -These commands are read-only and return deterministic exit codes for automation: - -- `0`: no warnings/errors -- `1`: warnings present, no errors -- `2`: errors present -- `64`: invalid CLI usage - -| Command | Description | -|---------|-------------| -| `inspect [--json] [--out ] [--include-pages]` | Full DB file/header/page integrity scan | -| `inspect-page [--json] [--hex]` | Inspect one page and optionally include hex dump | -| `check-wal [--json]` | Validate WAL header/frames/salts/checksums | -| `check-indexes [--index ] [--sample ] [--json]` | Validate index catalog/root/table/column consistency | - -See [Storage Inspector](storage-inspector.md) for full rule IDs and JSON contract details. - ---- - -## Examples - -### Introspection - -``` -csdb> .tables - users - orders - products - -csdb> .schema users -CREATE TABLE users ( - id INTEGER PRIMARY KEY IDENTITY, - name TEXT NOT NULL, - age INTEGER -) - -csdb> .indexes users - idx_users_name ON users(name) UNIQUE - -csdb> .info - Tables: 3 - Indexes: 2 - Views: 1 - Triggers: 1 - Collections: 0 - Mode: WAL enabled, timing off, snapshot off -``` - -### Transactions - -``` -csdb> .begin -Transaction started. - -csdb> INSERT INTO users VALUES (10, 'Test', 99); -1 row(s) affected - -csdb> .rollback -Transaction rolled back. - -csdb> SELECT * FROM users WHERE id = 10; -0 row(s) returned -``` - -### File Execution - -``` -csdb> .read samples/ecommerce-store/schema.sql -Executing 84 statements... -84 succeeded, 0 failed. -``` - -### Snapshot Mode - -``` -csdb> .snapshot on -Snapshot mode enabled. - -csdb> SELECT COUNT(*) FROM users; -┌──────────┐ -│ COUNT(*) │ -├──────────┤ -│ 42 │ -└──────────┘ - -csdb> .snapshot off -Snapshot mode disabled. -``` - ---- - -## See Also - -- [Getting Started Tutorial](getting-started.md) — C# API walkthrough with code examples -- [REST API Reference](rest-api.md) — HTTP endpoint documentation -- [FAQ](faq.md) — Common setup and troubleshooting answers -- [Sample Datasets](../samples/README.md) — SQL scripts to load test data diff --git a/docs/getting-started.md b/docs/getting-started.md index 487266ab..3bad3e40 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -721,9 +721,9 @@ await using (var db = await Database.OpenAsync("persistent.db")) ## Next Steps - [Architecture Guide](architecture.md) — How the engine works layer by layer -- [Internals & Contributing](internals.md) — How to extend the engine, testing strategy -- [REST API Reference](rest-api.md) — Use CSharpDB over HTTP from any language -- [MCP Server Reference](mcp-server.md) — Connect AI assistants to your database -- [CLI Reference](cli.md) — Interactive REPL with meta-commands +- [Internals & Contributing](https://csharpdb.com/docs/internals.html) — How to extend the engine, testing strategy +- [REST API Reference](https://csharpdb.com/docs/rest-api.html) — Use CSharpDB over HTTP from any language +- [MCP Server Reference](https://csharpdb.com/docs/mcp-server.html) — Connect AI assistants to your database +- [CLI Reference](https://csharpdb.com/docs/cli.html) — Interactive REPL with meta-commands - [FAQ](faq.md) — Common setup and troubleshooting answers - [Roadmap](roadmap.md) — Planned features and project direction diff --git a/docs/internals.md b/docs/internals.md deleted file mode 100644 index 69cbe664..00000000 --- a/docs/internals.md +++ /dev/null @@ -1,396 +0,0 @@ -# CSharpDB Internals & Contributing - -This guide is for developers who want to understand, extend, or contribute to CSharpDB. - -## Project Structure - -``` -CSharpDB.slnx -├── src/ -│ ├── CSharpDB.Primitives/ Shared types (no dependencies) -│ │ ├── DbType.cs Data type enum -│ │ ├── DbValue.cs Discriminated union value type -│ │ ├── Schema.cs ColumnDefinition, TableSchema, IndexSchema, TriggerSchema -│ │ └── CSharpDbException.cs Typed exception with ErrorCode -│ │ -│ ├── CSharpDB.Storage/ On-disk storage (depends on Primitives) -│ │ ├── PageConstants.cs Page size, header offsets, page types, WAL constants -│ │ ├── Varint.cs LEB128 variable-length integer codec -│ │ ├── IStorageDevice.cs Abstract async file I/O interface -│ │ ├── FileStorageDevice.cs Concrete implementation via RandomAccess -│ │ ├── Pager.cs Page cache, dirty tracking, allocation, transactions, WAL, snapshots -│ │ ├── WriteAheadLog.cs WAL file I/O: frames, commit, rollback, checkpoint, recovery -│ │ ├── WalIndex.cs In-memory WAL index + immutable snapshots for concurrent reads -│ │ ├── SlottedPage.cs Structured access to slotted page layout -│ │ ├── BTree.cs B+tree: insert, delete, find, split -│ │ ├── BTreeCursor.cs Forward-only cursor for scans and seeks -│ │ ├── RecordEncoder.cs Encode/decode DbValue[] ↔ byte[] -│ │ ├── SchemaSerializer.cs Encode/decode TableSchema ↔ byte[] -│ │ └── SchemaCatalog.cs In-memory schema cache (tables, indexes, views, triggers) backed by B+trees -│ │ -│ ├── CSharpDB.Storage.Diagnostics/ Read-only diagnostics (depends on Storage, Primitives) -│ │ ├── DatabaseInspector.cs Database file validation and page inspection -│ │ ├── WalInspector.cs WAL validation and frame inspection -│ │ ├── IndexInspector.cs Index integrity verification -│ │ └── README.md Package guide and usage examples -│ │ -│ ├── CSharpDB.Sql/ SQL frontend (depends on Primitives) -│ │ ├── TokenType.cs Token type enum -│ │ ├── Token.cs Token struct -│ │ ├── Tokenizer.cs Hand-rolled lexical scanner -│ │ ├── Ast.cs Statement and Expression AST nodes -│ │ ├── Parser.cs Recursive descent parser -│ │ ├── SqlScriptSplitter.cs Multi-statement script splitting (tracks BEGIN/END depth for triggers) -│ │ └── SqlStatementClassifier.cs Classifies statements as read-only or mutating -│ │ -│ ├── CSharpDB.Execution/ Query execution (depends on Primitives, Sql, Storage) -│ │ ├── IOperator.cs Iterator interface -│ │ ├── Operators.cs TableScan, IndexScan, Filter, Projection, Sort, Limit, Aggregate, Join -│ │ ├── ExpressionEvaluator.cs Evaluates Expression AST against a row -│ │ └── QueryPlanner.cs AST → operator tree or direct DML/DDL execution -│ │ -│ ├── CSharpDB.Engine/ Public API (depends on all above) -│ │ └── Database.cs Open, Execute, Transactions, Checkpoint, ReaderSession -│ │ -│ ├── CSharpDB.Client/ Unified client SDK (depends on Engine, Sql, Storage.Diagnostics) -│ │ ├── ICSharpDbClient.cs Public client contract (all database operations) -│ │ ├── CSharpDbClient.cs Factory: Create() → transport-specific implementation -│ │ ├── CSharpDbClientOptions.cs Configuration (DataSource, Endpoint, Transport, ConnectionString) -│ │ ├── CSharpDbTransport.cs Transport enum (Direct, Http, Grpc, NamedPipes) -│ │ ├── ServiceCollectionExtensions.cs DI registration (AddCSharpDbClient) -│ │ ├── Internal/ Transport resolver and direct/HTTP/gRPC implementations -│ │ └── Models/ Schema, data, procedure, transaction, and collection models -│ │ -│ ├── CSharpDB.Data/ ADO.NET provider (depends on Client + Engine) -│ │ ├── CSharpDbConnection.cs DbConnection implementation -│ │ ├── CSharpDbCommand.cs DbCommand with parameterized queries -│ │ ├── CSharpDbDataReader.cs DbDataReader with typed accessors -│ │ ├── CSharpDbParameter.cs DbParameter / DbParameterCollection -│ │ ├── CSharpDbTransaction.cs DbTransaction implementation -│ │ ├── CSharpDbFactory.cs DbProviderFactory registration -│ │ ├── RemoteDatabaseSession.cs Client-backed provider session for direct and gRPC access -│ │ ├── SqlParameterBinder.cs @param placeholder binding -│ │ ├── SharedMemoryDatabaseHost.cs Process-local named shared-memory host -│ │ └── TypeMapper.cs CSharpDB ↔ CLR type mapping -│ │ -│ ├── CSharpDB.Native/ NativeAOT C FFI library (depends on Engine, Execution, Primitives) -│ │ ├── NativeExports.cs 20 exported C functions (open, close, execute, result iteration, transactions, errors) -│ │ ├── HandleTable.cs GCHandle-based opaque pointer management -│ │ ├── StringCache.cs Unmanaged UTF-8 string lifetime management -│ │ ├── BlobCache.cs Pinned byte[] lifetime management -│ │ ├── ErrorState.cs Thread-local errno-style error reporting -│ │ └── csharpdb.h C header file for consumers -│ │ -│ ├── CSharpDB.Cli/ Interactive REPL (depends on Client) -│ │ ├── Program.cs Entry point with CLI argument parsing -│ │ ├── CliShellOptions.cs Parses --endpoint, --server, --transport flags -│ │ ├── Repl.cs Read-eval-print loop -│ │ ├── TableFormatter.cs ASCII table output with alignment -│ │ ├── MetaCommands.cs .tables, .schema, .quit, etc. -│ │ └── MetaCommandContext.cs Session state (client, transactions, snapshots) -│ │ -│ ├── CSharpDB.Admin/ Blazor Server admin dashboard (depends on Client) -│ │ ├── Program.cs Blazor Server entry point -│ │ ├── Services/ Theme, toast, modal, tab manager, DatabaseChangeService -│ │ └── Components/ Razor components for UI -│ │ -│ ├── CSharpDB.Api/ REST API (depends on Client) -│ │ ├── Program.cs ASP.NET Core Minimal API entry point -│ │ ├── Endpoints/ TableEndpoints, RowEndpoints, IndexEndpoints, ViewEndpoints, etc. -│ │ ├── Dtos/ Request/response record types -│ │ ├── Helpers/ JSON coercion helpers -│ │ └── Middleware/ Exception handling middleware -│ │ -│ ├── CSharpDB.Daemon/ gRPC host (depends on Client) -│ │ ├── Program.cs ASP.NET Core gRPC entry point -│ │ ├── Grpc/ Generated-contract host implementation -│ │ ├── Configuration/ Daemon config binding helpers -│ │ └── README.md Host model, deployment, and client usage -│ │ -│ └── CSharpDB.Mcp/ MCP server for AI assistants (depends on Client) -│ ├── Program.cs Generic Host with stdio transport -│ ├── Tools/ SchemaTools, DataTools, MutationTools, SqlTools (15 tools) -│ └── Helpers/ JSON serialization and value coercion -│ -├── clients/ -│ └── node/ Node.js/TypeScript client (wraps CSharpDB.Native via koffi) -│ ├── src/index.ts Database class, query/execute/transaction API -│ ├── src/native.ts koffi FFI bindings to CSharpDB.Native -│ ├── examples/ Basic usage example -│ └── tests/ Integration tests -│ -├── tests/ -│ ├── CSharpDB.Tests/ Engine unit + integration tests -│ │ ├── VarintTests.cs Varint round-trip encoding -│ │ ├── RecordEncoderTests.cs Row encoding/decoding -│ │ ├── TokenizerTests.cs SQL tokenization -│ │ ├── ParserTests.cs SQL parsing to AST -│ │ ├── IntegrationTests.cs Full SQL round-trips (end-to-end) -│ │ ├── WalTests.cs WAL mode: commit, rollback, crash recovery, snapshots -│ │ ├── ClientSqlExecutionTests.cs Client SDK SQL execution tests -│ │ └── SqlScriptSplitterTests.cs Script splitting edge cases -│ │ -│ ├── CSharpDB.Data.Tests/ ADO.NET provider tests -│ │ ├── ConnectionTests.cs Connection open/close/state -│ │ ├── CommandTests.cs Parameterized queries, ExecuteScalar, ExecuteNonQuery -│ │ ├── DataReaderTests.cs Typed getters, schema table, null handling -│ │ ├── TransactionTests.cs ADO.NET transaction commit/rollback -│ │ └── RemoteGrpcConnectionTests.cs ADO.NET over daemon-backed gRPC transport -│ │ -│ ├── CSharpDB.Cli.Tests/ CLI smoke + integration tests -│ │ -│ ├── CSharpDB.Api.Tests/ REST API transport and endpoint tests -│ │ -│ ├── CSharpDB.Daemon.Tests/ gRPC daemon transport tests -│ │ -│ └── CSharpDB.Benchmarks/ Performance benchmarks -│ -├── docs/ -│ ├── tutorials/native-ffi/ FFI tutorials (JavaScript via koffi, Python via ctypes) -│ ├── tutorials/storage/ Storage tutorial track, study examples, and advanced standalone examples -│ ├── roadmap.md Product roadmap and status -│ └── rest-api.md REST host reference -│ -└── samples/ Sample datasets + import helpers - ├── ecommerce-store/ - │ ├── schema.sql Northwind Electronics - │ └── procedures.json - ├── medical-clinic/ - │ ├── schema.sql Riverside Health Center - │ └── procedures.json - ├── school-district/ - │ ├── schema.sql Maplewood School District - │ └── procedures.json - ├── feature-tour/ - │ ├── schema.sql Northstar Field Services - │ ├── procedures.json - │ └── queries.sql - └── run-sample.csx -``` - ---- - -## How a SELECT Query Flows Through the System - -Tracing `db.ExecuteAsync("SELECT name FROM users WHERE age > 25")`: - -### Step 1: Database.ExecuteAsync (Engine) -- `Database.ExecuteAsync` calls `Parser.Parse(sql)` to get an AST -- Since it's a SELECT (not DML), no auto-transaction is started -- Calls `QueryPlanner.ExecuteAsync(stmt)` - -### Step 2: Parser.Parse (Sql) -- `Tokenizer.Tokenize()` scans the string into tokens: - `SELECT`, `name`, `FROM`, `users`, `WHERE`, `age`, `>`, `25` -- `Parser.ParseStatement()` recognizes SELECT and delegates to `ParseSelect()` -- Produces a `SelectStatement` with: - - Columns: `[ColumnRefExpression("name")]` - - TableName: `"users"` - - Where: `BinaryExpression(GreaterThan, ColumnRef("age"), Literal(25))` - -### Step 3: QueryPlanner.ExecuteSelect (Execution) -- Resolves `"users"` against `SchemaCatalog` to get `TableSchema` and root page -- Creates a `BTree` using the planner's own `Pager` (important for snapshot readers) -- Checks for usable indexes (equality predicates on indexed columns) -- Builds operator pipeline: - 1. `TableScanOperator(tree, schema)` — or `IndexScanOperator` if an index applies - 2. `FilterOperator(scan, whereExpr, schema)` — applies `age > 25` - 3. `ProjectionOperator(filter, [name], schema)` — extracts the `name` column -- Returns `QueryResult(projectionOp)` - -### Step 4: User iterates rows (Execution → Storage) -- Calling `result.GetRowsAsync()` opens the operator chain -- `ProjectionOperator.MoveNextAsync()` calls `FilterOperator.MoveNextAsync()` -- `FilterOperator` calls `TableScanOperator.MoveNextAsync()` in a loop -- `TableScanOperator` uses `BTreeCursor.MoveNextAsync()` to walk leaf pages -- `BTreeCursor` calls `Pager.GetPageAsync()` which checks: cache → WAL index → disk -- For each row, `FilterOperator` evaluates `ExpressionEvaluator.Evaluate(whereExpr, row, schema)` -- If the expression returns truthy, the row passes through to `ProjectionOperator` -- `ProjectionOperator` extracts the `name` column and yields it - ---- - -## How to Add a New SQL Statement - -Example: adding `TRUNCATE TABLE name`. - -### 1. Add AST node (`Ast.cs`) -```csharp -public sealed class TruncateTableStatement : Statement -{ - public required string TableName { get; init; } -} -``` - -### 2. Add parsing (`Parser.cs`) -Add `"TRUNCATE"` to the keyword list in `Tokenizer.cs`, then in `Parser.ParseStatement()`: -```csharp -TokenType.Truncate => ParseTruncate(), -``` -Implement `ParseTruncate()` to consume `TRUNCATE TABLE `. - -### 3. Add execution (`QueryPlanner.cs`) -```csharp -TruncateTableStatement truncate => await ExecuteTruncateAsync(truncate, ct), -``` -Implement `ExecuteTruncateAsync` — delete all rows from the B+tree, return `new QueryResult(deletedCount)`. - -### 4. Add tests (`IntegrationTests.cs`) -Write a test that creates a table, inserts rows, truncates, and verifies the table is empty. - ---- - -## How to Add a New Operator - -Example: adding a `DistinctOperator` that deduplicates rows. - -### 1. Create the operator class (`Operators.cs`) -```csharp -public sealed class DistinctOperator : IOperator -{ - private readonly IOperator _source; - private readonly HashSet _seen = new(); - - public ColumnDefinition[] OutputSchema => _source.OutputSchema; - public DbValue[] Current => _source.Current; - - public DistinctOperator(IOperator source) => _source = source; - - public ValueTask OpenAsync(CancellationToken ct) => _source.OpenAsync(ct); - - public async ValueTask MoveNextAsync(CancellationToken ct) - { - while (await _source.MoveNextAsync(ct)) - { - var key = string.Join("|", _source.Current.Select(v => v.ToString())); - if (_seen.Add(key)) return true; - } - return false; - } - - public ValueTask DisposeAsync() => _source.DisposeAsync(); -} -``` - -### 2. Wire it into the planner (`QueryPlanner.cs`) -Insert `DistinctOperator` into the operator tree in `ExecuteSelect()` when the AST indicates DISTINCT. - -### 3. Update the parser -Recognize `SELECT DISTINCT` in `ParseSelect()` and set a flag on `SelectStatement`. - ---- - -## Testing Strategy - -### Unit Tests (per layer) - -| Test file | Layer | What it tests | -|-----------|-------|---------------| -| `VarintTests.cs` | Storage | Varint encoding round-trips for unsigned/signed values | -| `RecordEncoderTests.cs` | Storage | Row encoding/decoding for all DbValue types | -| `TokenizerTests.cs` | Sql | Keyword recognition, string escaping, operators, numbers, comments | -| `ParserTests.cs` | Sql | AST generation for each statement type, complex expressions | - -### Integration Tests (end-to-end) - -| Category | Examples | -|----------|----------| -| **Basic CRUD** | CREATE TABLE + INSERT + SELECT, UPDATE, DELETE, DROP TABLE | -| **Filtering** | WHERE with AND/OR/NOT, LIKE, IN, BETWEEN, IS NULL | -| **Aggregates** | COUNT, SUM, AVG, MIN, MAX, GROUP BY, HAVING | -| **JOINs** | INNER JOIN, LEFT JOIN, RIGHT JOIN, CROSS JOIN, multi-table | -| **Schema** | ALTER TABLE (ADD/DROP/RENAME COLUMN, RENAME TABLE) | -| **Indexes** | CREATE INDEX, UNIQUE index, index-based lookups | -| **Views** | CREATE VIEW, SELECT from view, DROP VIEW | -| **CTEs** | WITH clause, multiple CTEs, CTE referencing CTE | -| **Triggers** | BEFORE/AFTER INSERT/UPDATE/DELETE, trigger with multiple statements | -| **Transactions** | BEGIN/COMMIT/ROLLBACK, persistence across reopen | -| **WAL** | Commit through WAL, rollback, crash recovery, concurrent readers, checkpointing | - -### ADO.NET Tests - -| Test file | What it tests | -|-----------|---------------| -| `ConnectionTests.cs` | Open/close, connection state, connection string parsing, GetTableNames/GetTableSchema | -| `CommandTests.cs` | ExecuteNonQuery, ExecuteScalar, ExecuteReader, parameterized queries | -| `DataReaderTests.cs` | Typed getters (GetInt64, GetString, GetDouble, GetBoolean), IsDBNull, GetSchemaTable, HasRows | -| `TransactionTests.cs` | ADO.NET transaction commit/rollback | -| `RemoteGrpcConnectionTests.cs` | Daemon-backed `Transport=Grpc;Endpoint=...` provider coverage | - -### Running Tests - -```bash -dotnet test tests/CSharpDB.Tests/CSharpDB.Tests.csproj --filter "FullyQualifiedName~IntegrationTests" -dotnet test tests/CSharpDB.Tests/CSharpDB.Tests.csproj --filter "FullyQualifiedName~WalTests" -dotnet test tests/CSharpDB.Data.Tests/CSharpDB.Data.Tests.csproj -dotnet test tests/CSharpDB.Cli.Tests/CSharpDB.Cli.Tests.csproj -dotnet test tests/CSharpDB.Api.Tests/CSharpDB.Api.Tests.csproj -dotnet test tests/CSharpDB.Daemon.Tests/CSharpDB.Daemon.Tests.csproj -``` - ---- - -## Concurrency Model - -CSharpDB now supports **concurrent readers plus isolated multi-writer execution -paths above a still-serialized storage commit boundary**: - -- **Storage boundary**: the pager/WAL layer still serializes the physical commit path with a writer lock. Dirty-page publication and durable WAL flush are not fully parallel at the storage layer. -- **Explicit multi-writer API**: `WriteTransaction` / `RunWriteTransactionAsync(...)` create isolated pager, catalog, and planner state per attempt, then detect conflicts and retry when needed. -- **Shared auto-commit work**: `UPDATE`, `DELETE`, and DDL already run through isolated write-transaction state internally on a shared `Database` handle. Shared auto-commit `INSERT` stays serialized by default and only switches to isolated write-transaction commits when `ImplicitInsertExecutionMode = ConcurrentWriteTransactions`. -- **Readers**: each reader acquires a `WalSnapshot`, so read traffic still sees a consistent point-in-time view while writers progress. -- **Checkpoint**: committed WAL pages are copied back to the main DB file during checkpoint. Checkpoint finalization still respects active snapshots that retain WAL frames. -- **Crash Recovery**: on open, the WAL is scanned for committed transactions and replayed into the durable view. - -``` -Legacy explicit flow: BeginTransaction → modify pages → CommitAsync → release writer gate -Multi-writer flow: WriteTransaction attempt → isolated state → conflict check → commit or retry -Reader flow: AcquireSnapshot → create snapshot pager → read pages (WAL or DB file) → release -Checkpoint flow: Acquire checkpoint state → copy WAL pages to DB → reset retained WAL when safe -``` - ---- - -## Current Limitations - -These are known simplifications: - -| Area | Limitation | -|------|-----------| -| **Functions** | Very limited built-in scalar function surface today; broader built-ins and user-defined functions remain planned | -| **Query** | Scalar/`IN`/`EXISTS` subqueries are supported, including correlated cases in `WHERE`, non-aggregate projection, and `UPDATE`/`DELETE`; correlated subqueries are still unsupported in `JOIN ON`, `GROUP BY`, `HAVING`, `ORDER BY`, and aggregate projections | -| **Query** | `UNION`, `INTERSECT`, and `EXCEPT` are supported; `UNION ALL` is not implemented yet | -| **Query** | No window functions | -| **Schema** | No SQL `DEFAULT` column values or `CHECK` constraints yet. Foreign keys are currently v1 only: single-column, column-level `REFERENCES` with optional `ON DELETE CASCADE`; table-level/composite/deferred foreign keys and `ON UPDATE` actions are not implemented | -| **Storage** | No page-level compression | -| **Storage** | No at-rest encryption for database/WAL files | -| **Storage** | Memory-mapped reads are opt-in and currently apply only to clean main-file pages; WAL-backed reads still rely on the WAL/cache path | -| **RowId** | Legacy table schemas without persisted high-water metadata may still pay a one-time key scan on first insert | -| **Collections** | `FindByIndexAsync` supports declared field-equality lookups; `FindAsync` remains a full scan | -| **Collections** | No JSON-path querying or expression/path-based document indexes yet | -| **Networking** | The shipping model still splits remote access between `CSharpDB.Api` for HTTP and `CSharpDB.Daemon` for gRPC; host consolidation plus named pipes remain planned | -| **Security** | Remote HTTP and gRPC deployment still rely on external network controls or front-end TLS termination; built-in auth, authorization, and TLS/mTLS support remain planned | -| **Concurrency** | The physical WAL commit path is still serialized at the storage boundary. Multi-writer gains depend on conflict shape, and shared auto-commit `INSERT` uses the legacy serialized path unless `ImplicitInsertExecutionMode = ConcurrentWriteTransactions` is enabled | -| **Indexes** | Composite indexes are supported, but ordered range-scan pushdown is still limited to narrower index shapes | - ---- - -## Roadmap - -See [docs/roadmap.md](roadmap.md) for the full roadmap with near-term, mid-term, and long-term goals. - -## See Also - -- [Architecture Guide](architecture.md) — Layer-by-layer design deep dive -- [Getting Started Tutorial](getting-started.md) — Step-by-step walkthrough with code examples -- [Client SDK](../src/CSharpDB.Client/README.md) — Unified client API and transport model -- [Native Library Reference](../src/CSharpDB.Native/README.md) — C FFI API, build instructions, cross-language examples -- [Node.js Client](../clients/node/README.md) — TypeScript/JavaScript package -- [REST API Reference](rest-api.md) — All 34 API endpoints with examples -- [MCP Server Reference](mcp-server.md) — AI assistant integration via Model Context Protocol -- [CLI Reference](cli.md) — Interactive REPL commands and meta-commands -- [Daemon Host Guide](../src/CSharpDB.Daemon/README.md) — gRPC host architecture, deployment, and client usage -- [Storage Tutorial Track](tutorials/storage/README.md) — Storage architecture, extensibility, and advanced example apps -- [FFI Tutorials](tutorials/native-ffi/README.md) — JavaScript and Python interop guides -- [Roadmap](roadmap.md) — Planned features and project direction -- [Benchmark Suite](../tests/CSharpDB.Benchmarks/README.md) — Performance data and comparison diff --git a/docs/mcp-server.md b/docs/mcp-server.md deleted file mode 100644 index 7d29af5d..00000000 --- a/docs/mcp-server.md +++ /dev/null @@ -1,235 +0,0 @@ -# MCP Server Reference - -CSharpDB includes a [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server that lets AI assistants — Claude Desktop, OpenAI Desktop, Cursor, VS Code Copilot, and others — interact with a CSharpDB database through a standardized tool interface. - -The server exposes 14 tools for schema inspection, data browsing, row mutations, and arbitrary SQL execution. - ---- - -## Running the Server - -```bash -# Default direct client target (ConnectionStrings:CSharpDB or csharpdb.db) -dotnet run --project src/CSharpDB.Mcp - -# Direct client target via database path -dotnet run --project src/CSharpDB.Mcp -- --database mydata.db - -# Explicit endpoint selection -dotnet run --project src/CSharpDB.Mcp -- --endpoint http://localhost:61818 --transport http -``` - -### Client Target Configuration - -The MCP server resolves `CSharpDbClientOptions` in this order: - -| Setting | Priority | Example | -|---------|----------|---------| -| `Endpoint` | `--endpoint` / `-e`, then `CSHARPDB_ENDPOINT` | `--endpoint http://localhost:61818` | -| `Transport` | `--transport` / `-t`, then `CSHARPDB_TRANSPORT` | `--transport http` | -| Direct database path | `--database` / `-d`, then `CSHARPDB_DATABASE` | `--database mydata.db` | -| Connection string | `ConnectionStrings:CSharpDB` from `appsettings.json` | `Data Source=csharpdb.db` | -| Default | `csharpdb.db` when no other target is supplied | `Data Source=csharpdb.db` | - -`Direct` remains the default transport. If you pass an HTTP endpoint without `--transport`, the client infers `Http`. `Http` and `Grpc` are implemented; named pipes remain the only additional transport option and are not implemented yet. - ---- - -## Client Configuration - -### Claude Desktop - -Add to `claude_desktop_config.json`: - -```json -{ - "mcpServers": { - "csharpdb": { - "command": "dotnet", - "args": ["run", "--project", "/path/to/src/CSharpDB.Mcp", "--", "--database", "/path/to/mydata.db"] - } - } -} -``` - -### Claude Code - -Add to `.mcp.json` in your project root: - -```json -{ - "mcpServers": { - "csharpdb": { - "command": "dotnet", - "args": ["run", "--project", "/path/to/src/CSharpDB.Mcp", "--", "--database", "/path/to/mydata.db"] - } - } -} -``` - -### Cursor / VS Code - -Add to your MCP settings (typically `.cursor/mcp.json` or VS Code settings): - -```json -{ - "mcpServers": { - "csharpdb": { - "command": "dotnet", - "args": ["run", "--project", "/path/to/src/CSharpDB.Mcp", "--", "--database", "/path/to/mydata.db"] - } - } -} -``` - -### OpenAI Codex CLI - -Add to your [Codex config](https://developers.openai.com/codex/mcp/) (`~/.codex/config.toml` globally, or `.codex/config.toml` per-project): - -```toml -[mcp_servers.csharpdb] -command = "dotnet" -args = ["run", "--project", "/path/to/src/CSharpDB.Mcp", "--", "--database", "/path/to/mydata.db"] -``` - -### ChatGPT Desktop - -ChatGPT Desktop currently only supports [remote MCP servers over HTTPS](https://developers.openai.com/apps-sdk/deploy/connect-chatgpt/) — it cannot spawn local stdio processes. To connect CSharpDB: - -1. Run the MCP server with an HTTP-to-stdio bridge such as [supergateway](https://github.com/nicholasgasior/supergateway) or [mcp-proxy](https://github.com/nicholasgasior/mcp-proxy): - ```bash - npx supergateway --port 8080 -- dotnet run --project /path/to/src/CSharpDB.Mcp -- --database /path/to/mydata.db - ``` -2. Expose the local port via [ngrok](https://ngrok.com/) or similar: - ```bash - ngrok http 8080 - ``` -3. In ChatGPT, go to **Settings → Connectors → Create** and paste the public HTTPS URL. - -> **Tip:** If you only need SQL access from ChatGPT, the [REST API](rest-api.md) at `http://localhost:61818` may be simpler — no bridge required. - -### LM Studio - -LM Studio supports MCP servers starting from [v0.3.17](https://lmstudio.ai/blog/lmstudio-v0.3.17). Open the **Program** tab in the right-hand sidebar, click **Install → Edit mcp.json**, and add CSharpDB: - -```json -{ - "mcpServers": { - "csharpdb": { - "command": "dotnet", - "args": ["run", "--project", "/path/to/src/CSharpDB.Mcp", "--", "--database", "/path/to/mydata.db"] - } - } -} -``` - -The `mcp.json` file lives at: - -| OS | Path | -|----|------| -| Windows | `%USERPROFILE%\.lmstudio\mcp.json` | -| macOS / Linux | `~/.lmstudio/mcp.json` | - -> **Note:** For all stdio-based clients (Claude, Codex, Cursor, LM Studio), make sure `dotnet` is available on your system PATH. - -![LM Studio integration with CSharpDB](images/LMStudioIntegration.png) - ---- - -## Available Tools - -### Schema Tools - -| Tool | Description | Parameters | -|------|-------------|------------| -| `GetDatabaseInfo` | Get the database file path | — | -| `ListTables` | List all table names | — | -| `DescribeTable` | Get column names, types, and constraints | `tableName` | -| `ListIndexes` | List all indexes with table, columns, uniqueness | — | -| `ListViews` | List all views with their SQL definitions | — | -| `ListTriggers` | List all triggers with timing, event, and body | — | - -### Data Tools - -| Tool | Description | Parameters | -|------|-------------|------------| -| `BrowseTable` | Paginated row browsing with schema | `tableName`, `page?` (default 1), `pageSize?` (default 50) | -| `BrowseView` | Paginated view result browsing | `viewName`, `page?` (default 1), `pageSize?` (default 50) | -| `GetRowByPk` | Fetch a single row by primary key | `tableName`, `pkColumn`, `pkValue` | -| `GetRowCount` | Get total row count for a table | `tableName` | - -### Mutation Tools - -| Tool | Description | Parameters | -|------|-------------|------------| -| `InsertRow` | Insert a row into a table | `tableName`, `valuesJson` | -| `UpdateRow` | Update a row by primary key | `tableName`, `pkColumn`, `pkValue`, `valuesJson` | -| `DeleteRow` | Delete a row by primary key | `tableName`, `pkColumn`, `pkValue` | - -The `valuesJson` parameter accepts a JSON object string with column names as keys: - -```json -{"name": "Alice", "age": 30, "email": "alice@example.com"} -``` - -Values are automatically coerced to CSharpDB types: integers become `INTEGER`, decimals become `REAL`, strings become `TEXT`, null stays `NULL`. - -`DescribeTable` and `BrowseTable` include `isIdentity` metadata for identity columns. - -### SQL Tool - -| Tool | Description | Parameters | -|------|-------------|------------| -| `ExecuteSql` | Execute any SQL statement | `sql` | - -This is the general-purpose tool for anything not covered by the specialized tools — DDL (`CREATE TABLE`, `ALTER TABLE`, `DROP`), complex queries with JOINs and aggregates, `CREATE INDEX`, `CREATE VIEW`, `CREATE TRIGGER`, and so on. - ---- - -## Example Conversations - -Once connected, an AI assistant can interact with CSharpDB naturally: - -> **User:** What tables are in the database? -> -> **Assistant:** *(calls ListTables)* The database has 7 tables: customers, categories, products, orders, order_items, reviews, and shipping_addresses. - -> **User:** Show me the top 5 customers by order total. -> -> **Assistant:** *(calls ExecuteSql with a JOIN + GROUP BY query)* Here are the top 5 customers... - -> **User:** Add a new product called "Widget Pro" priced at $29.99 in category 2. -> -> **Assistant:** *(calls InsertRow)* Done — inserted 1 row into the products table. - ---- - -## Architecture - -The MCP server is a .NET console app using stdio transport: - -``` -AI Client (Claude, Cursor, etc.) - │ - │ stdio (JSON-RPC) - │ -CSharpDB.Mcp (Generic Host) - │ -ICSharpDbClient - │ -CSharpDB.Client (transport-selecting SDK) - │ -CSharpDB Engine (B+tree, WAL, Pager) - │ -mydata.db -``` - -All 14 tools are thin wrappers around `ICSharpDbClient`. In the current repo, MCP uses the direct engine-backed client by default, but the host can also be pointed at an HTTP endpoint through the same client options. The MCP SDK (`ModelContextProtocol` NuGet package) handles protocol framing, tool discovery, and stdio transport. - ---- - -## See Also - -- [Getting Started Tutorial](getting-started.md) — Learn the CSharpDB API step by step -- [REST API Reference](rest-api.md) — HTTP endpoints for the same operations -- [CLI Reference](cli.md) — Interactive REPL for manual exploration diff --git a/docs/releases/v3.4.0-pr-notes.md b/docs/releases/v3.4.0-pr-notes.md new file mode 100644 index 00000000..2ef8c881 --- /dev/null +++ b/docs/releases/v3.4.0-pr-notes.md @@ -0,0 +1,82 @@ +## Summary + +This PR updates the `v3.4.0` docs/site work so the website is now the +authoritative published home for the verified high-confidence docs set, while +still preserving the original long-form markdown content for the source-heavy +guides that had only been partially migrated before. + +The website work adds companion source-reference HTML pages for the original +architecture, getting-started, performance, query-execution-pipeline, SQL, +storage-engine, roadmap, and SQLite benchmarking markdown guides. The shorter +curated pages remain in place, but now link to the full source references when +readers need the original complete material. The docs cleanup also removes the +duplicate markdown copies of the CLI, REST API, MCP server, internals, and +storage inspector guides after their website versions were verified. + +This PR also adds a new blog post covering the C# launcher pattern for +`CSharpDB.Admin` and keeps the post wired into the blog index and sitemap. + +## Type of Change + +- [ ] Bug fix +- [ ] New feature +- [ ] Breaking change +- [x] Documentation update +- [x] Refactor / maintenance +- [ ] Tests only + +## Related Issues + +No issue numbers were linked for this docs/site cleanup. Included work in this +PR: + +- audited the current `docs/` markdown tree against the published `www` site +- added source-reference HTML pages for the partially migrated long-form docs +- linked the curated pages to those full references and updated the sitemap +- removed high-confidence duplicated markdown files after link cleanup +- added the `CSharpDB.Admin` C# launcher blog post to the website + +## Testing + +- [ ] `dotnet build CSharpDB.slnx` +- [ ] Relevant tests executed +- [ ] Failure-path tests executed (if applicable: cancellation, invalid/unsupported inputs, non-`DbException` paths) +- [x] Manual verification performed (if applicable) + +Validation performed for this PR: + +- markdown-to-HTML content audit confirmed: + - `docs/architecture.md` -> `www/architecture-reference.html` + - `docs/getting-started.md` -> `www/docs/getting-started-reference.html` + - `docs/performance.md` -> `www/docs/performance-reference.html` + - `docs/query-execution-pipeline.md` -> `www/docs/query-execution-pipeline.html` + - `docs/sql-reference.md` -> `www/docs/sql-reference.html` + - `docs/storage/README.md` -> `www/docs/storage-engine-reference.html` + - `docs/roadmap.md` -> `www/roadmap-reference.html` + - `docs/query-and-durable-write-performance/csharpdb-vs-sqlite-benchmarking-blog.md` -> `www/blog/csharpdb-vs-sqlite-benchmarking-reference.html` +- coverage audit showed 100% heading coverage and 99.7-100% token overlap for + the migrated source-reference set +- `python -c "import xml.etree.ElementTree as ET; ET.parse('www/sitemap.xml')"` + passed after updating the sitemap +- repo scan found no remaining references to the deleted duplicated markdown + files (`cli.md`, `rest-api.md`, `mcp-server.md`, `internals.md`, + `storage-inspector.md`) + +## Checklist + +- [x] I followed the project style and conventions. +- [x] I updated docs for user-facing changes. +- [x] I verified no sensitive data was added. + +## Notes for Reviewers + +- The important semantic correction in this range is that the original + `docs/query-execution-pipeline.md` content is no longer implicitly treated as + if it were the same page as the shipped ETL pipelines guide. It now has its + own source-reference route under `www/docs/query-execution-pipeline.html`. +- The shorter curated docs remain the primary public landing pages; the new + source-reference pages exist to preserve the original long-form markdown + material before deleting redundant copies from `docs/`. +- Only the high-confidence duplicated markdown files were removed in this pass. + The remaining markdown files still need their own migration/verification work + before they are safe to delete. diff --git a/docs/rest-api.md b/docs/rest-api.md deleted file mode 100644 index ffd83085..00000000 --- a/docs/rest-api.md +++ /dev/null @@ -1,635 +0,0 @@ -# CSharpDB REST API Reference - -The CSharpDB REST API exposes the full database feature set over HTTP, enabling cross-language interoperability. Built with ASP.NET Core Minimal APIs, it includes OpenAPI documentation and an interactive Scalar UI. - -## Running the API - -```bash -dotnet run --project src/CSharpDB.Api -``` - -The API starts on `http://localhost:61818` (HTTP) and `https://localhost:61819` (HTTPS). - -**Interactive documentation:** Open `http://localhost:61818/scalar/v1` in a browser to explore and test endpoints with the Scalar API explorer. - -## Configuration - -The default database path is configured in `src/CSharpDB.Api/appsettings.json`: - -```json -{ - "ConnectionStrings": { - "CSharpDB": "Data Source=csharpdb.db" - } -} -``` - -CORS is enabled for all origins by default (development convenience). JSON responses use camelCase naming and omit null values. - ---- - -## Endpoints - -All endpoints are prefixed with `/api`. - -### Database Info - -#### `GET /api/info` - -Returns a summary of the database. - -**Response:** -```json -{ - "dataSource": "csharpdb.db", - "tableCount": 3, - "indexCount": 2, - "viewCount": 1, - "triggerCount": 1, - "procedureCount": 2 -} -``` - ---- - -### Storage Inspection - -Read-only physical diagnostics endpoints for `.db` and `.wal` inspection. - -#### `GET /api/inspect` - -Run a full database file inspection. - -**Query parameters:** -- `includePages` (default: `false`) — include per-page decoded details in the response -- `path` (optional) — override database path for this request - -#### `GET /api/inspect/wal` - -Inspect WAL header/frame/checksum state. - -**Query parameters:** -- `path` (optional) — override database path for this request - -#### `GET /api/inspect/page/{id}` - -Inspect a single page by page id. - -**Query parameters:** -- `hex` (default: `false`) — include page hex dump -- `path` (optional) — override database path for this request - -#### `GET /api/inspect/indexes` - -Validate index metadata and root tree reachability. - -**Query parameters:** -- `index` (optional) — check one index by name -- `sample` (optional) — sample size hint for future index validation passes -- `path` (optional) — override database path for this request - -Responses follow the diagnostics models documented in [Storage Inspector](storage-inspector.md). - ---- - -### Maintenance - -#### `POST /api/maintenance/migrate-foreign-keys` - -Validate or apply foreign-key retrofit migration for older databases whose tables do not yet persist FK metadata. - -**Request:** -```json -{ - "validateOnly": true, - "backupDestinationPath": "pre-fk.backup.db", - "violationSampleLimit": 100, - "constraints": [ - { - "tableName": "orders", - "columnName": "customer_id", - "referencedTableName": "customers", - "referencedColumnName": "id", - "onDelete": "Restrict" - } - ] -} -``` - -**Response:** -```json -{ - "validateOnly": true, - "succeeded": false, - "backupDestinationPath": null, - "affectedTables": 1, - "appliedForeignKeys": 1, - "copiedRows": 0, - "violationCount": 1, - "violations": [ - { - "tableName": "orders", - "columnName": "customer_id", - "referencedTableName": "customers", - "referencedColumnName": "id", - "childKeyColumnName": "id", - "childKeyValue": 42, - "childValue": 999, - "reason": "MissingReferencedParent" - } - ], - "appliedConstraints": [ - { - "tableName": "orders", - "columnName": "customer_id", - "referencedTableName": "customers", - "referencedColumnName": "id", - "constraintName": "fk_orders_customer_id_abcd1234", - "supportingIndexName": "__fk_orders_customer_id_abcd1234", - "onDelete": "Restrict" - } - ] -} -``` - -Notes: -- `validateOnly = true` previews the migration without mutating schema or data. -- `backupDestinationPath` is optional and is only used during apply mode. -- Paths are resolved on the API host machine, not on the caller. - ---- - -### Tables - -#### `GET /api/tables` - -List all table names. - -**Response:** -```json -["users", "orders", "products"] -``` - -#### `GET /api/tables/{name}/schema` - -Get the full schema for a table. - -**Response:** -```json -{ - "tableName": "users", - "columns": [ - { "name": "id", "type": "Integer", "nullable": false, "isPrimaryKey": true, "isIdentity": true }, - { "name": "name", "type": "Text", "nullable": false, "isPrimaryKey": false, "isIdentity": false }, - { "name": "age", "type": "Integer", "nullable": true, "isPrimaryKey": false, "isIdentity": false } - ] -} -``` - -#### `GET /api/tables/{name}/count` - -Get the row count for a table. - -**Response:** -```json -{ "count": 42 } -``` - -#### `DELETE /api/tables/{name}` - -Drop a table. - -**Response:** `200 OK` with `{ "message": "Table 'users' dropped." }` - -#### `PATCH /api/tables/{name}/rename` - -Rename a table. - -**Request:** -```json -{ "newName": "customers" } -``` - -**Response:** `200 OK` with `{ "message": "Table renamed from 'users' to 'customers'." }` - -#### `POST /api/tables/{name}/columns` - -Add a column to a table. - -**Request:** -```json -{ "columnName": "email", "type": "TEXT", "notNull": false } -``` - -**Response:** `200 OK` with `{ "message": "Column 'email' added to 'users'." }` - -#### `DELETE /api/tables/{name}/columns/{col}` - -Drop a column. - -**Response:** `200 OK` with `{ "message": "Column 'email' dropped from 'users'." }` - -#### `PATCH /api/tables/{name}/columns/{col}/rename` - -Rename a column. - -**Request:** -```json -{ "newName": "full_name" } -``` - -**Response:** `200 OK` with `{ "message": "Column renamed from 'name' to 'full_name' in 'users'." }` - ---- - -### Rows - -#### `GET /api/tables/{name}/rows` - -Browse table rows with pagination. - -**Query parameters:** -- `page` (default: 1) — Page number -- `pageSize` (default: 50, max: 1000) — Rows per page - -**Response:** -```json -{ - "tableName": "users", - "page": 1, - "pageSize": 50, - "totalRows": 3, - "columns": ["id", "name", "age"], - "rows": [ - { "id": 1, "name": "Alice", "age": 30 }, - { "id": 2, "name": "Bob", "age": 25 } - ] -} -``` - -#### `GET /api/tables/{name}/rows/{pkValue}` - -Get a single row by primary key. - -**Query parameters:** -- `pkColumn` (default: "id") — Name of the primary key column - -**Response:** -```json -{ "id": 1, "name": "Alice", "age": 30 } -``` - -#### `POST /api/tables/{name}/rows` - -Insert a new row. - -**Request:** -```json -{ "values": { "id": 4, "name": "Diana", "age": 28 } } -``` - -**Response:** `201 Created` with `{ "message": "Row inserted into 'users'.", "rowsAffected": 1 }` - -#### `PUT /api/tables/{name}/rows/{pkValue}` - -Update a row by primary key. - -**Query parameters:** -- `pkColumn` (default: "id") — Name of the primary key column - -**Request:** -```json -{ "values": { "name": "Diana Updated", "age": 29 } } -``` - -**Response:** `200 OK` with `{ "message": "Row updated in 'users'.", "rowsAffected": 1 }` - -#### `DELETE /api/tables/{name}/rows/{pkValue}` - -Delete a row by primary key. - -**Query parameters:** -- `pkColumn` (default: "id") — Name of the primary key column - -**Response:** `200 OK` with `{ "message": "Row deleted from 'users'.", "rowsAffected": 1 }` - ---- - -### Indexes - -#### `GET /api/indexes` - -List all indexes. - -**Response:** -```json -[ - { "name": "idx_category", "tableName": "products", "columnName": "category", "isUnique": false } -] -``` - -#### `POST /api/indexes` - -Create an index. - -**Request:** -```json -{ "indexName": "idx_email", "tableName": "users", "columnName": "email", "isUnique": true } -``` - -**Response:** `201 Created` with `{ "message": "Index 'idx_email' created." }` - -#### `PUT /api/indexes/{name}` - -Update (drop and recreate) an index. - -**Request:** -```json -{ "newIndexName": "idx_user_email", "tableName": "users", "columnName": "email", "isUnique": true } -``` - -**Response:** `200 OK` - -#### `DELETE /api/indexes/{name}` - -Drop an index. - -**Response:** `200 OK` with `{ "message": "Index 'idx_email' dropped." }` - ---- - -### Views - -#### `GET /api/views` - -List all views. - -**Response:** -```json -["order_summary", "product_catalog"] -``` - -#### `GET /api/views/{name}` - -Get a view definition. - -**Response:** -```json -{ - "viewName": "order_summary", - "selectSql": "SELECT o.id, c.name, o.total FROM orders o INNER JOIN customers c ON o.customer_id = c.id" -} -``` - -#### `GET /api/views/{name}/rows` - -Browse view results with pagination. - -**Query parameters:** -- `page` (default: 1) -- `pageSize` (default: 50, max: 1000) - -**Response:** Same shape as table browse (columns + rows). - -#### `POST /api/views` - -Create a view. - -**Request:** -```json -{ "viewName": "expensive_products", "selectSql": "SELECT name, price FROM products WHERE price > 50" } -``` - -**Response:** `201 Created` - -#### `PUT /api/views/{name}` - -Update a view (drop and recreate). - -**Request:** -```json -{ "newViewName": "expensive_products", "selectSql": "SELECT name, price FROM products WHERE price > 100" } -``` - -**Response:** `200 OK` - -#### `DELETE /api/views/{name}` - -Drop a view. - -**Response:** `200 OK` - ---- - -### Triggers - -#### `GET /api/triggers` - -List all triggers. - -**Response:** -```json -[ - { - "name": "trg_update_stock", - "tableName": "order_items", - "timing": "After", - "event": "Insert", - "bodySql": "UPDATE products SET stock = stock - NEW.quantity WHERE id = NEW.product_id" - } -] -``` - -#### `POST /api/triggers` - -Create a trigger. - -**Request:** -```json -{ - "triggerName": "trg_audit_insert", - "tableName": "users", - "timing": "After", - "event": "Insert", - "bodySql": "INSERT INTO audit_log VALUES ('INSERT', NEW.name)" -} -``` - -Timing values: `"Before"`, `"After"` -Event values: `"Insert"`, `"Update"`, `"Delete"` - -**Response:** `201 Created` - -#### `PUT /api/triggers/{name}` - -Update a trigger (drop and recreate). - -**Response:** `200 OK` - -#### `DELETE /api/triggers/{name}` - -Drop a trigger. - -**Response:** `200 OK` - ---- - -### SQL Execution - -#### `POST /api/sql/execute` - -Execute an arbitrary SQL statement. - -**Request:** -```json -{ "sql": "SELECT name, price FROM products WHERE price > 10 ORDER BY price DESC" } -``` - -**Response (query):** -```json -{ - "isQuery": true, - "columnNames": ["name", "price"], - "rows": [ - { "name": "Widget", "price": 29.99 }, - { "name": "Gadget", "price": 14.99 } - ], - "rowsAffected": 0, - "elapsed": 1.23 -} -``` - -**Response (mutation):** -```json -{ - "isQuery": false, - "columnNames": null, - "rows": null, - "rowsAffected": 3, - "elapsed": 0.87 -} -``` - ---- - -### Procedures - -Table-backed procedure catalog (`__procedures`) with strict parameter metadata validation and transactional execution. - -#### `GET /api/procedures` - -List procedure metadata. - -#### `GET /api/procedures/{name}` - -Get one procedure definition. - -#### `POST /api/procedures` - -Create a procedure. - -**Request:** -```json -{ - "name": "GetUserById", - "bodySql": "SELECT * FROM users WHERE id = @id;", - "parameters": [ - { "name": "id", "type": "INTEGER", "required": true, "default": null, "description": "User ID" } - ], - "description": "Lookup user by ID", - "isEnabled": true -} -``` - -#### `PUT /api/procedures/{name}` - -Update (or rename) a procedure. - -**Request:** -```json -{ - "newName": "GetUserById", - "bodySql": "SELECT * FROM users WHERE id = @id;", - "parameters": [ - { "name": "id", "type": "INTEGER", "required": true } - ], - "description": "Updated description", - "isEnabled": true -} -``` - -#### `DELETE /api/procedures/{name}` - -Delete a procedure. - -#### `POST /api/procedures/{name}/execute` - -Execute a stored procedure by name. - -**Request:** -```json -{ - "args": { - "id": 123 - } -} -``` - -**Response (success):** -```json -{ - "procedureName": "GetUserById", - "succeeded": true, - "statements": [ - { - "statementIndex": 0, - "statementText": "SELECT * FROM users WHERE id = @id;", - "isQuery": true, - "columnNames": ["id", "name"], - "rows": [{ "id": 123, "name": "Alice" }], - "rowsAffected": 1, - "elapsedMs": 0.34 - } - ], - "error": null, - "failedStatementIndex": null, - "elapsedMs": 0.51 -} -``` - -**Response (validation/runtime failure):** -`400 Bad Request` with the same shape and `succeeded = false`. - ---- - -## Error Handling - -The API uses standard HTTP status codes and returns structured error responses: - -| HTTP Status | CSharpDB Error Code | Meaning | -|-------------|---------------------|---------| -| 400 | `SyntaxError`, `TypeMismatch` | Bad request — invalid SQL or type mismatch | -| 404 | `TableNotFound`, `ColumnNotFound` | Resource not found | -| 409 | `DuplicateKey`, `TableAlreadyExists` | Conflict — duplicate resource | -| 422 | `ConstraintViolation` | Constraint violated (NOT NULL, UNIQUE) | -| 503 | `Busy` | Database is busy (another writer is active) | -| 500 | (other) | Unexpected server error | - -**Error response format:** -```json -{ - "error": "Table 'nonexistent' not found.", - "code": "TableNotFound" -} -``` - -In development mode, a `detail` field with a stack trace is included. - ---- - -## See Also - -- [Getting Started Tutorial](getting-started.md) — Engine API walkthrough -- [Architecture Guide](architecture.md) — How the engine works internally -- [CLI Reference](cli.md) — Interactive REPL commands -- [Sample Datasets](../samples/README.md) — Ready-to-run SQL scripts diff --git a/docs/roadmap.md b/docs/roadmap.md index a714eea0..bb217aa7 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,6 +1,6 @@ # CSharpDB Roadmap -This document outlines the planned direction for CSharpDB, organized by timeframe and priority. Items are roughly ordered by expected impact within each tier, and statuses are intended to reflect the current `v3.0.0` state of the repo. +This document outlines the planned direction for CSharpDB, organized by timeframe and priority. Items are roughly ordered by expected impact within each tier, and statuses are intended to reflect the current `v3.4.0` state of the repo. --- @@ -30,7 +30,7 @@ Recently completed improvements to query performance, storage/runtime behavior, | **Table/index statistics** | ANALYZE command with persisted row counts, column NDV/min/max, stale tracking, and initial stats-guided index selection in the query planner | Done | | **Collection binary payloads** | Binary direct-payload format with faster hydration, direct field/path extraction, and richer path-based indexing | Done | | **Collection path indexes** | Nested scalar, array-element, nested array-object, Guid, temporal, and ordered text path indexes with `FindByPathAsync` / `FindByPathRangeAsync` | Done | -| **Hybrid storage mode** | Lazy-resident durable storage with gRPC tunable file-cache configuration | Done | +| **Hybrid storage mode** | Lazy-resident durable storage with gRPC tunable file-cache configuration; Admin direct local hosting keeps a warm in-process database instance and uses hybrid incremental-durable options by default | Done | | **Client backup/restore** | `BackupAsync` / `RestoreAsync` as first-class `ICSharpDbClient` operations across direct, HTTP, gRPC, CLI, and Admin | Done | | **Older DB foreign-key retrofit migration** | Validate/apply maintenance workflow that rewrites existing child tables with persisted FK metadata across direct, HTTP, gRPC, CLI, and Admin | Done | @@ -49,10 +49,10 @@ SQL feature parity, provider/tooling compatibility, and ecosystem expansion. | **`DEFAULT` column values** | Allow default expressions in column definitions | Planned | | **`CHECK` constraints** | Arbitrary expression-based constraints per column or per table | Planned | | **Foreign key constraints** | v1 support for single-column, column-level `REFERENCES` with optional `ON DELETE CASCADE`, plus `sys.foreign_keys` and metadata/tooling surfaces | Done | -| **Remote host consolidation** | Fold the current `CSharpDB.Api` REST/HTTP surface into `CSharpDB.Daemon` so one long-running server host can serve REST, gRPC, and future local transports from a shared warm `Database` instance | Planned | +| **Remote host consolidation** | `CSharpDB.Daemon` now hosts the existing REST/HTTP `/api` surface and gRPC from one long-running process backed by the same warm daemon-hosted client; standalone `CSharpDB.Api` remains supported for REST-only hosting | Done | | **Remote host security** | Add built-in authentication, authorization, and transport-security options for remote HTTP and gRPC access, including API keys, protected admin endpoints, and TLS/mTLS deployment support | Planned | -| **Daemon service packaging** | Package the existing `CSharpDB.Daemon` host as a persistent background service across systemd, Windows Service, and launchd | Planned | -| **Cross-platform deployment** | dotnet tool, self-contained binaries, Docker, Homebrew, winget, install scripts | Planned | +| **Daemon service packaging** | Package the existing `CSharpDB.Daemon` host as a persistent background service across systemd, Windows Service, and launchd | Done | +| **Cross-platform deployment** | Self-contained daemon archives and install scripts ship for Windows, Linux, and macOS; dotnet tool, Docker, Homebrew, and winget distribution remain future work | In Progress | | **NuGet package** | Publish and maintain `CSharpDB.Engine`, `CSharpDB.Data`, `CSharpDB.Client`, and `CSharpDB.Primitives` as the primary NuGet packages | Done | | **Connection pooling** | Pool underlying direct embedded sessions behind `CSharpDbConnection` to amortize open/close cost | Done | | **Admin dashboard improvements** | Richer SQL editor UX, query history, deeper diagnostics, and integrated Forms/Reports tooling beyond the core schema/procedure/storage surface | Done | @@ -102,7 +102,7 @@ These are known simplifications in the current implementation: | **Indexes** | Equality lookups support current `INTEGER`/`TEXT` indexes, but ordered range-scan pushdown is still limited to single-column `INTEGER` index paths | | **RowId** | Legacy table schemas without persisted high-water metadata may pay a one-time key scan on first insert | | **Collections** | `FindByIndexAsync` supports declared field-equality lookups; `FindByPathAsync` and `FindByPathRangeAsync` support path-based queries on indexed paths; `FindAsync` remains a full scan for unindexed predicates | -| **Networking** | The current shipping model still splits remote access between `CSharpDB.Api` for HTTP and `CSharpDB.Daemon` for gRPC; host consolidation plus named pipes remain planned and are not implemented yet | +| **Networking** | `CSharpDB.Daemon` now hosts both REST and gRPC from one process; named pipes remain reserved but are not implemented end to end today | | **Security** | Remote HTTP and gRPC deployment still rely on external network controls or front-end TLS termination; built-in authentication, authorization, and TLS/mTLS support are still planned | | **Text / Multilingual** | Text is stored as UTF-8 and supports all Unicode languages; default semantics remain ordinal, but opt-in `BINARY`, `NOCASE`, `NOCASE_AI`, and `ICU:` collation are implemented for SQL and collection indexes. Dedicated ordered SQL text index optimization remains planned | | **Concurrency** | The physical WAL commit path is still serialized at the storage boundary. Initial multi-writer support is shipped, but observed gains still depend on conflict shape and whether shared auto-commit `INSERT` is left on the default serialized path | @@ -149,6 +149,7 @@ Major features already implemented: - Collection secondary field indexes via `EnsureIndexAsync` / `FindByIndexAsync` - Maintenance report, `REINDEX`, and `VACUUM` flows across client, CLI, API, and Admin UI - Dedicated `CSharpDB.Daemon` gRPC host for remote `CSharpDB.Client` access +- Remote host consolidation in `CSharpDB.Daemon`, with REST `/api` and gRPC sharing the same warm hosted database client - Storage tuning presets, bounded WAL read caching, memory-mapped main-file reads, and sliced background WAL auto-checkpointing - SQL executor/read-path fast paths for compact projections, broader join/index coverage, grouped aggregates, and correlated subquery filters - Batch-first SQL row-batch execution foundation with batch-aware scan/index/join roots, shared predicate/projection kernels, and batch-native generic aggregate paths @@ -174,7 +175,7 @@ Major features already implemented: ## See Also - [Architecture Guide](architecture.md) — How the engine is structured -- [Internals & Contributing](internals.md) — How to extend the engine +- [Internals & Contributing](https://csharpdb.com/docs/internals.html) — How to extend the engine - [Deployment & Installation Plan](deployment/README.md) — Cross-platform distribution via dotnet tool, Docker, Homebrew, winget, and install scripts - [Multi-Writer Follow-Up Plan](multi-writer-follow-up-plan.md) — Post-initial multi-writer roadmap, insert-path gaps, and release criteria for broader completion - [Query And Durable Write Performance Plan](query-and-durable-write-performance/README.md) — Combined optimizer phase-2 plus durable-write completion plan, shipped state, and remaining benchmark/future-work boundaries diff --git a/docs/storage-inspector.md b/docs/storage-inspector.md deleted file mode 100644 index ded0a112..00000000 --- a/docs/storage-inspector.md +++ /dev/null @@ -1,202 +0,0 @@ -# Storage Inspector - -The Storage Inspector is a read-only diagnostics toolkit for understanding the physical state of a CSharpDB database (`.db`) and WAL (`.db.wal`) file. - -It is CLI-first and deterministic: - -- no data mutation -- machine-readable JSON output (`schemaVersion: "1.0"`) -- fixed exit codes for automation - -## Scope (v1) - -Diagnostics cover: - -- database file header and page map -- B+tree page and cell structure checks -- WAL header/frame/checksum checks -- catalog and index-root reachability checks - -Out of scope: - -- repair/auto-fix -- compaction/vacuum -- on-disk format changes - -## Commands - -Run commands through the CLI command mode: - -```bash -csharpdb inspect [--json] [--out ] [--include-pages] -csharpdb inspect-page [--json] [--hex] -csharpdb check-wal [--json] -csharpdb check-indexes [--index ] [--sample ] [--json] -``` - -`inspect` supports `--out` to write JSON to a file. - -## Client/API/Admin - -Client SDK methods (`CSharpDB.Client`): - -- `InspectStorageAsync(path?, includePages?)` -- `CheckWalAsync(path?)` -- `InspectPageAsync(pageId, includeHex?, path?)` -- `CheckIndexesAsync(path?, indexName?, sampleSize?)` - -REST endpoints (`CSharpDB.Api`): - -- `GET /api/inspect` -- `GET /api/inspect/wal` -- `GET /api/inspect/page/{id}` -- `GET /api/inspect/indexes` - -Admin UI (`CSharpDB.Admin`): - -- Sidebar `Storage` action opens the **Storage** tab -- Displays header summary, page histogram, index checks, issue list, and page drill-down -- Also exposes maintenance actions plus backup/restore, with paths resolved on the connected host - -## Exit Codes - -- `0`: no warnings/errors -- `1`: warnings present, no errors -- `2`: one or more errors present -- `64`: invalid CLI usage/arguments - -## JSON Contract - -All inspector JSON responses include: - -- `schemaVersion` (current: `"1.0"`) - -The top-level report object depends on the command: - -- `inspect` -> `DatabaseInspectReport` -- `inspect-page` -> `PageInspectReport` -- `check-wal` -> `WalInspectReport` -- `check-indexes` -> `IndexInspectReport` - -`issues` entries use: - -- `code` (stable rule identifier) -- `severity` (`Info`, `Warning`, `Error`) -- `message` -- optional `pageId` -- optional `offset` - -## Integrity Rule IDs (v1) - -Database/header/page checks: - -- `DB_HEADER_SHORT` -- `DB_HEADER_BAD_MAGIC` -- `DB_HEADER_BAD_VERSION` -- `DB_HEADER_BAD_PAGE_SIZE` -- `DB_PAGE_COUNT_MISMATCH` -- `DB_FILE_TRAILING_BYTES` -- `DB_PAGE_SHORT_READ` -- `PAGE_HEADER_OUT_OF_RANGE` -- `PAGE_CELL_COUNT_OVERFLOW` -- `PAGE_CELL_CONTENT_START_OOB` -- `PAGE_CELL_CONTENT_OVERLAP` -- `PAGE_DUPLICATE_CELL_POINTER` -- `PAGE_CELL_POINTER_OOB` -- `CELL_VARINT_INVALID` -- `CELL_HEADER_INVALID` -- `CELL_TOTAL_SIZE_OOB` -- `LEAF_CELL_PAYLOAD_TOO_SMALL` -- `LEAF_CELL_KEY_OOB` -- `LEAF_CELL_PAYLOAD_OOB` -- `INTERIOR_CELL_PAYLOAD_TOO_SMALL` -- `INTERIOR_CELL_BYTES_OOB` -- `BTREE_LEAF_KEY_ORDER` -- `PAGE_TYPE_UNKNOWN` - -B+tree reachability/cross checks: - -- `SCHEMA_ROOT_MISSING` -- `BTREE_CHILD_OUT_OF_RANGE` -- `BTREE_PAGE_MISSING` -- `BTREE_PAGE_TYPE_INVALID` -- `BTREE_NULL_CHILD_REFERENCE` -- `CATALOG_ROOT_OUT_OF_RANGE` -- `CATALOG_ROOT_MISSING` -- `CATALOG_ROOT_BAD_PAGE_TYPE` -- `CATALOG_ENTRY_PAYLOAD_SHORT` -- `CATALOG_TABLE_SCHEMA_DECODE_FAILED` -- `CATALOG_INDEX_ENTRY_PAYLOAD_SHORT` -- `CATALOG_INDEX_SCHEMA_DECODE_FAILED` -- `CATALOG_VIEW_SCHEMA_DECODE_FAILED` -- `CATALOG_TRIGGER_SCHEMA_DECODE_FAILED` -- `INDEX_NOT_FOUND` -- `INDEX_ROOT_OUT_OF_RANGE` -- `INDEX_ROOT_MISSING` -- `INDEX_ROOT_BAD_PAGE_TYPE` -- `INDEX_TABLE_MISSING` -- `INDEX_COLUMN_MISSING` - -WAL checks: - -- `WAL_HEADER_SHORT` -- `WAL_HEADER_BAD_MAGIC` -- `WAL_HEADER_BAD_VERSION` -- `WAL_HEADER_BAD_PAGE_SIZE` -- `WAL_TRAILING_PARTIAL_FRAME` -- `WAL_FRAME_HEADER_SHORT` -- `WAL_FRAME_PAGE_SHORT` -- `WAL_FRAME_SALT_MISMATCH` -- `WAL_FRAME_HEADER_CHECKSUM_MISMATCH` -- `WAL_FRAME_DATA_CHECKSUM_MISMATCH` -- `WAL_NO_COMMIT_MARKER` - -## Examples - -Human-readable summary: - -```bash -csharpdb inspect mydata.db -``` - -Machine-readable JSON: - -```bash -csharpdb inspect mydata.db --json -``` - -Write JSON report to file: - -```bash -csharpdb inspect mydata.db --json --out inspect-report.json -``` - -Inspect one page including hex dump: - -```bash -csharpdb inspect-page mydata.db 12 --hex -``` - -Check WAL state: - -```bash -csharpdb check-wal mydata.db -``` - -Check all indexes: - -```bash -csharpdb check-indexes mydata.db -``` - -Check one index: - -```bash -csharpdb check-indexes mydata.db --index idx_users_name --json -``` - -## Troubleshooting - -- If `check-wal` reports no WAL file, that is usually normal after clean close/checkpoint. -- If DB checks report page-count mismatch with no errors, run `check-wal` to determine whether recent committed data still resides in WAL. -- If checks run while a writer is active, warnings can appear due to concurrent file changes; rerun diagnostics after write activity stops. diff --git a/samples/README.md b/samples/README.md index c7fdc373..bbb32933 100644 --- a/samples/README.md +++ b/samples/README.md @@ -306,5 +306,5 @@ The `native-ffi/` folder contains wrappers and examples for calling CSharpDB fro - [Getting Started Tutorial](../docs/getting-started.md) - [CSharpDB.Client README](../src/CSharpDB.Client/README.md) -- [REST API Reference](../docs/rest-api.md) -- [CLI Reference](../docs/cli.md) +- [REST API Reference](https://csharpdb.com/docs/rest-api.html) +- [CLI Reference](https://csharpdb.com/docs/cli.html) diff --git a/src/CSharpDB.Admin/Helpers/SqlCompletionProvider.cs b/src/CSharpDB.Admin/Helpers/SqlCompletionProvider.cs new file mode 100644 index 00000000..23bb7d24 --- /dev/null +++ b/src/CSharpDB.Admin/Helpers/SqlCompletionProvider.cs @@ -0,0 +1,651 @@ +using System.Text.RegularExpressions; + +namespace CSharpDB.Admin.Helpers; + +public enum SqlCompletionSourceKind +{ + Table, + View, + SystemCatalog, +} + +public enum SqlCompletionSuggestionKind +{ + Keyword, + Source, + Column, + Procedure, + Function, +} + +public sealed record SqlCompletionSource(string Name, SqlCompletionSourceKind Kind); + +public sealed record SqlCompletionColumn(string Name, string? Type, string SourceName); + +public sealed class SqlCompletionCatalog +{ + public static readonly SqlCompletionCatalog Empty = new(); + + public IReadOnlyList Sources { get; init; } = []; + + public IReadOnlyDictionary> ColumnsBySource { get; init; } = + new Dictionary>(StringComparer.OrdinalIgnoreCase); + + public IReadOnlyList Procedures { get; init; } = []; + + public IReadOnlyList GetColumnsForSource(string sourceName) + => ColumnsBySource.TryGetValue(sourceName, out var columns) ? columns : []; +} + +public sealed record SqlCompletionSuggestion( + string Label, + string InsertText, + string Detail, + SqlCompletionSuggestionKind Kind, + int ReplacementStart, + int ReplacementEnd, + int CaretOffset) +{ + public int CaretPosition => ReplacementStart + CaretOffset; +} + +public sealed record SqlCompletionResult(IReadOnlyList Suggestions) +{ + public static readonly SqlCompletionResult Empty = new([]); +} + +public static partial class SqlCompletionProvider +{ + private const int MaxSuggestions = 12; + + private static readonly string[] s_columnContextPreviousTokens = + [ + "WHERE", "AND", "OR", "ON", "BY", "HAVING", "SET" + ]; + + private static readonly SqlCompletionKeyword[] s_keywords = + [ + new("SELECT", "SELECT ", "query rows"), + new("FROM", "FROM ", "choose source"), + new("WHERE", "WHERE ", "filter rows"), + new("SET", "SET ", "assign columns"), + new("ORDER BY", "ORDER BY ", "sort rows"), + new("GROUP BY", "GROUP BY ", "group rows"), + new("HAVING", "HAVING ", "filter groups"), + new("LIMIT", "LIMIT ", "limit rows"), + new("OFFSET", "OFFSET ", "skip rows"), + new("JOIN", "JOIN ", "join source"), + new("LEFT JOIN", "LEFT JOIN ", "join optional source"), + new("INSERT INTO", "INSERT INTO ", "insert rows"), + new("VALUES", "VALUES ", "provide row values"), + new("UPDATE", "UPDATE ", "update rows"), + new("DELETE FROM", "DELETE FROM ", "delete rows"), + new("CREATE TABLE", "CREATE TABLE ", "create table"), + new("CREATE INDEX", "CREATE INDEX ", "create index"), + new("EXEC", "EXEC ", "execute procedure"), + ]; + + private static readonly SqlCompletionKeyword[] s_functions = + [ + new("COUNT", "COUNT()", "aggregate"), + new("SUM", "SUM()", "aggregate"), + new("AVG", "AVG()", "aggregate"), + new("MIN", "MIN()", "aggregate"), + new("MAX", "MAX()", "aggregate"), + new("COALESCE", "COALESCE()", "function"), + new("LOWER", "LOWER()", "function"), + new("UPPER", "UPPER()", "function"), + new("LENGTH", "LENGTH()", "function"), + ]; + + public static SqlCompletionResult GetCompletions( + string sql, + int caret, + SqlCompletionCatalog catalog, + bool explicitTrigger = false) + { + sql ??= string.Empty; + catalog ??= SqlCompletionCatalog.Empty; + caret = Math.Clamp(caret, 0, sql.Length); + + if (IsInsideSingleQuotedString(sql, caret)) + return SqlCompletionResult.Empty; + + var token = ReadCurrentToken(sql, caret); + if (TryGetDotColumnCompletions(sql, caret, token, catalog, out var dotResult)) + return MaybeSuppressExactMatch(dotResult, token.Prefix, explicitTrigger); + + if (TryGetSelectListCompletions(sql, caret, token, catalog, out var selectResult)) + return MaybeSuppressExactMatch(selectResult, token.Prefix, explicitTrigger); + + string? previousToken = ReadPreviousToken(sql, token.Start); + if (IsSourceContext(previousToken)) + return MaybeSuppressExactMatch( + BuildSourceSuggestions(catalog, token.Prefix, token.Start, caret, sourceForSelectList: false), + token.Prefix, + explicitTrigger); + + if (IsExecContext(previousToken)) + return MaybeSuppressExactMatch( + BuildProcedureSuggestions(catalog, token.Prefix, token.Start, caret), + token.Prefix, + explicitTrigger); + + if (TryGetInsertCompletions(sql, caret, token, catalog, out var insertResult)) + return MaybeSuppressExactMatch(insertResult, token.Prefix, explicitTrigger); + + if (previousToken is not null + && s_columnContextPreviousTokens.Contains(previousToken, StringComparer.OrdinalIgnoreCase) + && TryFindPrimarySource(sql, caret, out string sourceName)) + { + return MaybeSuppressExactMatch( + BuildColumnSuggestions(catalog, sourceName, token.Prefix, token.Start, caret), + token.Prefix, + explicitTrigger); + } + + if (TryGetUpdateSetCompletions(sql, caret, token, out var updateSetResult)) + return updateSetResult; + + if (token.Prefix.Length == 0 && !explicitTrigger) + return SqlCompletionResult.Empty; + + return MaybeSuppressExactMatch( + BuildKeywordSuggestions(token.Prefix, token.Start, caret, includeAllWhenEmpty: explicitTrigger), + token.Prefix, + explicitTrigger); + } + + private static SqlCompletionResult MaybeSuppressExactMatch( + SqlCompletionResult result, + string prefix, + bool explicitTrigger) + { + if (explicitTrigger || prefix.Length == 0 || result.Suggestions.Count == 0) + return result; + + return result.Suggestions.Any(suggestion => suggestion.Label.Equals(prefix, StringComparison.OrdinalIgnoreCase)) + ? SqlCompletionResult.Empty + : result; + } + + private static bool TryGetSelectListCompletions( + string sql, + int caret, + SqlCompletionToken token, + SqlCompletionCatalog catalog, + out SqlCompletionResult result) + { + result = SqlCompletionResult.Empty; + + if (!TryGetStatementBounds(sql, caret, out int statementStart, out int statementEnd)) + return false; + + string statement = sql[statementStart..statementEnd]; + int relativeCaret = caret - statementStart; + if (!TryFindSelectList(statement, relativeCaret, out int selectEnd)) + return false; + + string betweenSelectAndCaret = statement[selectEnd..relativeCaret]; + if (ContainsWholeWord(betweenSelectAndCaret, "FROM")) + return false; + + if (TryFindSourceAfterCaret(statement, relativeCaret, out string sourceAfterCaret)) + { + result = BuildColumnSuggestions(catalog, sourceAfterCaret, token.Prefix, token.Start, caret); + return result.Suggestions.Count > 0; + } + + result = BuildSourceSuggestions(catalog, token.Prefix, token.Start, caret, sourceForSelectList: true); + return result.Suggestions.Count > 0; + } + + private static bool TryGetDotColumnCompletions( + string sql, + int caret, + SqlCompletionToken token, + SqlCompletionCatalog catalog, + out SqlCompletionResult result) + { + result = SqlCompletionResult.Empty; + int dotOffset = token.Prefix.LastIndexOf('.'); + if (dotOffset >= 0) + { + string embeddedQualifier = token.Prefix[..dotOffset]; + string embeddedPrefix = token.Prefix[(dotOffset + 1)..]; + if (embeddedQualifier.Length == 0) + return false; + + string embeddedSourceName = ResolveSourceQualifier(sql, caret, embeddedQualifier); + result = BuildColumnSuggestions( + catalog, + embeddedSourceName, + embeddedPrefix, + token.Start + dotOffset + 1, + caret); + return result.Suggestions.Count > 0; + } + + if (token.Start <= 0 || sql[token.Start - 1] != '.') + return false; + + if (!TryReadIdentifierBefore(sql, token.Start - 1, out string qualifier)) + return false; + + string sourceName = ResolveSourceQualifier(sql, caret, qualifier); + result = BuildColumnSuggestions(catalog, sourceName, token.Prefix, token.Start, caret); + return result.Suggestions.Count > 0; + } + + private static SqlCompletionResult BuildKeywordSuggestions( + string prefix, + int replacementStart, + int replacementEnd, + bool includeAllWhenEmpty) + { + var suggestions = s_keywords + .Where(keyword => includeAllWhenEmpty || MatchesPrefix(keyword.Label, prefix)) + .Concat(s_functions.Where(function => includeAllWhenEmpty || MatchesPrefix(function.Label, prefix))) + .Select(keyword => + { + bool isFunction = s_functions.Contains(keyword); + string insertText = keyword.InsertText; + int caretOffset = isFunction && insertText.EndsWith("()", StringComparison.Ordinal) + ? insertText.Length - 1 + : insertText.Length; + return new SqlCompletionSuggestion( + keyword.Label, + insertText, + keyword.Detail, + isFunction ? SqlCompletionSuggestionKind.Function : SqlCompletionSuggestionKind.Keyword, + replacementStart, + replacementEnd, + caretOffset); + }) + .Take(MaxSuggestions) + .ToArray(); + + return suggestions.Length == 0 ? SqlCompletionResult.Empty : new SqlCompletionResult(suggestions); + } + + private static SqlCompletionResult BuildSourceSuggestions( + SqlCompletionCatalog catalog, + string prefix, + int replacementStart, + int replacementEnd, + bool sourceForSelectList) + { + var suggestions = catalog.Sources + .Where(source => MatchesPrefix(source.Name, prefix)) + .OrderBy(source => source.Kind) + .ThenBy(source => source.Name, StringComparer.OrdinalIgnoreCase) + .Take(MaxSuggestions) + .Select(source => + { + string insertText = sourceForSelectList + ? $"{Environment.NewLine}FROM {source.Name}" + : source.Name; + int caretOffset = sourceForSelectList ? 0 : insertText.Length; + return new SqlCompletionSuggestion( + source.Name, + insertText, + source.Kind switch + { + SqlCompletionSourceKind.View => "view", + SqlCompletionSourceKind.SystemCatalog => "system catalog", + _ => "table", + }, + SqlCompletionSuggestionKind.Source, + replacementStart, + replacementEnd, + caretOffset); + }) + .ToArray(); + + return suggestions.Length == 0 ? SqlCompletionResult.Empty : new SqlCompletionResult(suggestions); + } + + private static SqlCompletionResult BuildColumnSuggestions( + SqlCompletionCatalog catalog, + string sourceName, + string prefix, + int replacementStart, + int replacementEnd, + bool includeWildcard = true) + { + var columns = catalog.GetColumnsForSource(sourceName); + if (columns.Count == 0) + return SqlCompletionResult.Empty; + + var suggestions = new List(); + if (includeWildcard && (prefix.Length == 0 || MatchesPrefix("*", prefix))) + { + suggestions.Add(new SqlCompletionSuggestion( + "*", + "*", + sourceName, + SqlCompletionSuggestionKind.Column, + replacementStart, + replacementEnd, + 1)); + } + + suggestions.AddRange(columns + .Where(column => MatchesPrefix(column.Name, prefix)) + .OrderBy(column => column.Name, StringComparer.OrdinalIgnoreCase) + .Take(MaxSuggestions - suggestions.Count) + .Select(column => new SqlCompletionSuggestion( + column.Name, + column.Name, + column.Type is null ? column.SourceName : $"{column.SourceName} - {column.Type}", + SqlCompletionSuggestionKind.Column, + replacementStart, + replacementEnd, + column.Name.Length))); + + return suggestions.Count == 0 ? SqlCompletionResult.Empty : new SqlCompletionResult(suggestions); + } + + private static SqlCompletionResult BuildProcedureSuggestions( + SqlCompletionCatalog catalog, + string prefix, + int replacementStart, + int replacementEnd) + { + var suggestions = catalog.Procedures + .Where(name => MatchesPrefix(name, prefix)) + .OrderBy(name => name, StringComparer.OrdinalIgnoreCase) + .Take(MaxSuggestions) + .Select(name => new SqlCompletionSuggestion( + name, + name, + "procedure", + SqlCompletionSuggestionKind.Procedure, + replacementStart, + replacementEnd, + name.Length)) + .ToArray(); + + return suggestions.Length == 0 ? SqlCompletionResult.Empty : new SqlCompletionResult(suggestions); + } + + private static SqlCompletionToken ReadCurrentToken(string sql, int caret) + { + int start = caret; + while (start > 0 && IsIdentifierPart(sql[start - 1])) + start--; + + return new SqlCompletionToken(start, caret, sql[start..caret]); + } + + private static string? ReadPreviousToken(string sql, int beforeIndex) + { + int index = Math.Clamp(beforeIndex, 0, sql.Length); + while (index > 0 && char.IsWhiteSpace(sql[index - 1])) + index--; + + while (index > 0 && !IsIdentifierPart(sql[index - 1])) + index--; + + int end = index; + while (index > 0 && IsIdentifierPart(sql[index - 1])) + index--; + + return end > index ? sql[index..end] : null; + } + + private static bool TryFindSelectList(string statement, int relativeCaret, out int selectEnd) + { + selectEnd = -1; + foreach (Match match in SelectKeywordRegex().Matches(statement[..relativeCaret])) + selectEnd = match.Index + match.Length; + + return selectEnd >= 0 && relativeCaret >= selectEnd; + } + + private static bool TryFindSourceAfterCaret(string statement, int relativeCaret, out string sourceName) + { + sourceName = string.Empty; + var match = FromSourceRegex().Match(statement[relativeCaret..]); + if (!match.Success) + return false; + + sourceName = match.Groups["source"].Value; + return true; + } + + private static bool TryFindPrimarySource(string sql, int caret, out string sourceName) + { + sourceName = string.Empty; + if (!TryGetStatementBounds(sql, caret, out int statementStart, out int statementEnd)) + return false; + + string statement = sql[statementStart..statementEnd]; + + var match = UpdateSourceRegex().Match(statement); + if (!match.Success) + match = FromSourceRegex().Match(statement); + + if (!match.Success) + return false; + + sourceName = match.Groups["source"].Value; + return true; + } + + private static bool TryGetInsertCompletions( + string sql, + int caret, + SqlCompletionToken token, + SqlCompletionCatalog catalog, + out SqlCompletionResult result) + { + result = SqlCompletionResult.Empty; + if (!TryGetStatementBounds(sql, caret, out int statementStart, out _)) + return false; + + string beforeCaret = sql[statementStart..caret]; + if (TryGetInsertColumnCompletions(beforeCaret, token, catalog, out result)) + return true; + + return TryGetInsertValuesCompletions(beforeCaret, token, out result); + } + + private static bool TryGetInsertColumnCompletions( + string beforeCaret, + SqlCompletionToken token, + SqlCompletionCatalog catalog, + out SqlCompletionResult result) + { + result = SqlCompletionResult.Empty; + var match = InsertColumnListOpenRegex().Match(beforeCaret); + if (!match.Success) + return false; + + string sourceName = match.Groups["source"].Value; + result = BuildColumnSuggestions( + catalog, + sourceName, + token.Prefix, + token.Start, + token.End, + includeWildcard: false); + + return result.Suggestions.Count > 0; + } + + private static bool TryGetInsertValuesCompletions( + string beforeCaret, + SqlCompletionToken token, + out SqlCompletionResult result) + { + result = SqlCompletionResult.Empty; + var match = InsertColumnListClosedRegex().Match(beforeCaret); + if (!match.Success || !MatchesPrefix("VALUES", token.Prefix)) + return false; + + result = new SqlCompletionResult( + [ + new SqlCompletionSuggestion( + "VALUES", + "VALUES ", + "provide row values", + SqlCompletionSuggestionKind.Keyword, + token.Start, + token.End, + 7) + ]); + + return true; + } + + private static bool TryGetUpdateSetCompletions( + string sql, + int caret, + SqlCompletionToken token, + out SqlCompletionResult result) + { + result = SqlCompletionResult.Empty; + if (!TryGetStatementBounds(sql, caret, out int statementStart, out _)) + return false; + + string beforeCaret = sql[statementStart..caret]; + var match = UpdateSourceRegex().Match(beforeCaret); + if (!match.Success) + return false; + + string trailingText = beforeCaret[match.Length..]; + if (ContainsWholeWord(trailingText, "SET")) + return false; + + string trimmedTrailingText = trailingText.Trim(); + if (trimmedTrailingText.Length > 0 + && (!trimmedTrailingText.Equals(token.Prefix, StringComparison.Ordinal) + || !MatchesPrefix("SET", token.Prefix))) + { + return false; + } + + result = new SqlCompletionResult( + [ + new SqlCompletionSuggestion( + "SET", + "SET ", + "assign columns", + SqlCompletionSuggestionKind.Keyword, + token.Start, + caret, + 4) + ]); + + return true; + } + + private static string ResolveSourceQualifier(string sql, int caret, string qualifier) + { + if (!TryGetStatementBounds(sql, caret, out int statementStart, out int statementEnd)) + return qualifier; + + foreach (Match match in SourceWithAliasRegex().Matches(sql[statementStart..statementEnd])) + { + string source = match.Groups["source"].Value; + string alias = match.Groups["alias"].Success ? match.Groups["alias"].Value : string.Empty; + if (source.Equals(qualifier, StringComparison.OrdinalIgnoreCase) + || alias.Equals(qualifier, StringComparison.OrdinalIgnoreCase)) + { + return source; + } + } + + return qualifier; + } + + private static bool TryGetStatementBounds(string sql, int caret, out int start, out int end) + { + start = sql.LastIndexOf(';', Math.Max(0, caret - 1)) + 1; + int nextSemicolon = sql.IndexOf(';', caret); + end = nextSemicolon < 0 ? sql.Length : nextSemicolon; + return start >= 0 && end >= start; + } + + private static bool TryReadIdentifierBefore(string sql, int beforeDotIndex, out string identifier) + { + identifier = string.Empty; + int end = beforeDotIndex; + while (end > 0 && char.IsWhiteSpace(sql[end - 1])) + end--; + + int start = end; + while (start > 0 && IsIdentifierPart(sql[start - 1])) + start--; + + if (end <= start) + return false; + + identifier = sql[start..end]; + return true; + } + + private static bool IsSourceContext(string? previousToken) + => previousToken is not null + && (previousToken.Equals("FROM", StringComparison.OrdinalIgnoreCase) + || previousToken.Equals("JOIN", StringComparison.OrdinalIgnoreCase) + || previousToken.Equals("UPDATE", StringComparison.OrdinalIgnoreCase) + || previousToken.Equals("INTO", StringComparison.OrdinalIgnoreCase)); + + private static bool IsExecContext(string? previousToken) + => previousToken is not null + && (previousToken.Equals("EXEC", StringComparison.OrdinalIgnoreCase) + || previousToken.Equals("EXECUTE", StringComparison.OrdinalIgnoreCase)); + + private static bool ContainsWholeWord(string text, string word) + => Regex.IsMatch(text, $@"\b{Regex.Escape(word)}\b", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + + private static bool MatchesPrefix(string value, string prefix) + => prefix.Length == 0 || value.StartsWith(prefix, StringComparison.OrdinalIgnoreCase); + + private static bool IsIdentifierPart(char value) + => char.IsLetterOrDigit(value) || value == '_' || value == '.'; + + private static bool IsInsideSingleQuotedString(string sql, int caret) + { + bool inString = false; + for (int i = 0; i < caret; i++) + { + if (sql[i] != '\'') + continue; + + if (inString && i + 1 < caret && sql[i + 1] == '\'') + { + i++; + continue; + } + + inString = !inString; + } + + return inString; + } + + [GeneratedRegex(@"\bSELECT\b", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex SelectKeywordRegex(); + + [GeneratedRegex(@"\bFROM\s+(?[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)?)", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex FromSourceRegex(); + + [GeneratedRegex(@"^\s*INSERT\s+INTO\s+(?[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)?)\s*\((?[^)]*)$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex InsertColumnListOpenRegex(); + + [GeneratedRegex(@"^\s*INSERT\s+INTO\s+(?[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)?)\s*\((?[^)]*)\)\s*(?[A-Za-z_]*)$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex InsertColumnListClosedRegex(); + + [GeneratedRegex(@"^\s*UPDATE\s+(?[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)?)\b", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex UpdateSourceRegex(); + + [GeneratedRegex(@"\b(?:FROM|JOIN)\s+(?[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)?)(?:\s+(?:AS\s+)?(?[A-Za-z_][A-Za-z0-9_]*))?", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex SourceWithAliasRegex(); + + private sealed record SqlCompletionKeyword(string Label, string InsertText, string Detail); + + private sealed record SqlCompletionToken(int Start, int End, string Prefix); +} diff --git a/tests/CSharpDB.Admin.Forms.Tests/Components/Shared/SqlCompletionProviderTests.cs b/tests/CSharpDB.Admin.Forms.Tests/Components/Shared/SqlCompletionProviderTests.cs new file mode 100644 index 00000000..24486101 --- /dev/null +++ b/tests/CSharpDB.Admin.Forms.Tests/Components/Shared/SqlCompletionProviderTests.cs @@ -0,0 +1,244 @@ +using CSharpDB.Admin.Helpers; + +namespace CSharpDB.Admin.Forms.Tests.Components.Shared; + +public sealed class SqlCompletionProviderTests +{ + [Fact] + public void GetCompletions_TypedS_SuggestsSelect() + { + var result = SqlCompletionProvider.GetCompletions("s", 1, CreateCatalog()); + + var suggestion = Assert.Single(result.Suggestions, s => s.Label == "SELECT"); + Assert.Equal(0, suggestion.ReplacementStart); + Assert.Equal(1, suggestion.ReplacementEnd); + Assert.Equal("SELECT ", suggestion.InsertText); + } + + [Fact] + public void GetCompletions_TypedLi_SuggestsLimit() + { + var result = SqlCompletionProvider.GetCompletions("li", 2, CreateCatalog()); + + var suggestion = Assert.Single(result.Suggestions, s => s.Label == "LIMIT"); + Assert.Equal(0, suggestion.ReplacementStart); + Assert.Equal(2, suggestion.ReplacementEnd); + Assert.Equal("LIMIT ", suggestion.InsertText); + } + + [Fact] + public void GetCompletions_AfterUpdateTableTypingSe_SuggestsSet() + { + var result = SqlCompletionProvider.GetCompletions("UPDATE Customers se", "UPDATE Customers se".Length, CreateCatalog()); + + var suggestion = Assert.Single(result.Suggestions, s => s.Label == "SET"); + Assert.Equal("SET ", suggestion.InsertText); + } + + [Fact] + public void GetCompletions_AfterUpdateTableWithTrailingSpace_SuggestsSet() + { + string sql = "UPDATE Customers "; + + var result = SqlCompletionProvider.GetCompletions(sql, sql.Length, CreateCatalog()); + + var suggestion = Assert.Single(result.Suggestions, s => s.Label == "SET"); + Assert.Equal("SET ", suggestion.InsertText); + } + + [Fact] + public void GetCompletions_InsideInsertColumnList_SuggestsColumns() + { + string sql = "INSERT INTO Customers ("; + + var result = SqlCompletionProvider.GetCompletions(sql, sql.Length, CreateCatalog()); + + Assert.DoesNotContain(result.Suggestions, s => s.Label == "*"); + Assert.Contains(result.Suggestions, s => s.Label == "CustomerId"); + Assert.Contains(result.Suggestions, s => s.Label == "Name"); + Assert.Contains(result.Suggestions, s => s.Label == "Email"); + } + + [Fact] + public void GetCompletions_AfterInsertColumnComma_SuggestsColumns() + { + string sql = "INSERT INTO Customers (CustomerId, "; + + var result = SqlCompletionProvider.GetCompletions(sql, sql.Length, CreateCatalog()); + + Assert.Contains(result.Suggestions, s => s.Label == "Name"); + Assert.Contains(result.Suggestions, s => s.Label == "Email"); + } + + [Fact] + public void GetCompletions_AfterInsertColumnListWithTrailingSpace_SuggestsValues() + { + string sql = "INSERT INTO Customers (CustomerId, Name) "; + + var result = SqlCompletionProvider.GetCompletions(sql, sql.Length, CreateCatalog()); + + var suggestion = Assert.Single(result.Suggestions, s => s.Label == "VALUES"); + Assert.Equal("VALUES ", suggestion.InsertText); + } + + [Fact] + public void GetCompletions_AfterInsertColumnListTypingVa_SuggestsValues() + { + string sql = "INSERT INTO Customers (CustomerId, Name) va"; + + var result = SqlCompletionProvider.GetCompletions(sql, sql.Length, CreateCatalog()); + + var suggestion = Assert.Single(result.Suggestions, s => s.Label == "VALUES"); + Assert.Equal("VALUES ", suggestion.InsertText); + } + + [Fact] + public void GetCompletions_AfterSelectWithoutSource_SuggestsSources() + { + var result = SqlCompletionProvider.GetCompletions("SELECT ", 7, CreateCatalog()); + + Assert.Contains(result.Suggestions, s => s.Label == "Customers" && s.Detail == "table"); + Assert.Contains(result.Suggestions, s => s.Label == "CustomerSummary" && s.Detail == "view"); + } + + [Fact] + public void GetCompletions_SelectSourceSuggestionInsertsFromAndLeavesCaretInSelectList() + { + var result = SqlCompletionProvider.GetCompletions("SELECT ", 7, CreateCatalog()); + + var suggestion = Assert.Single(result.Suggestions, s => s.Label == "Customers"); + Assert.Equal($"{Environment.NewLine}FROM Customers", suggestion.InsertText); + Assert.Equal(7, suggestion.CaretPosition); + } + + [Fact] + public void GetCompletions_SelectListWithSourceAfterCaret_SuggestsColumns() + { + string sql = $"SELECT {Environment.NewLine}FROM Customers"; + + var result = SqlCompletionProvider.GetCompletions(sql, "SELECT ".Length, CreateCatalog()); + + Assert.Contains(result.Suggestions, s => s.Label == "*"); + Assert.Contains(result.Suggestions, s => s.Label == "CustomerId" && s.Detail == "Customers - INTEGER"); + Assert.Contains(result.Suggestions, s => s.Label == "Name" && s.Detail == "Customers - TEXT"); + } + + [Fact] + public void GetCompletions_FromContextFiltersSources() + { + var result = SqlCompletionProvider.GetCompletions("SELECT * FROM Cu", "SELECT * FROM Cu".Length, CreateCatalog()); + + Assert.Contains(result.Suggestions, s => s.Label == "Customers"); + Assert.Contains(result.Suggestions, s => s.Label == "CustomerSummary"); + Assert.DoesNotContain(result.Suggestions, s => s.Label == "Orders"); + } + + [Fact] + public void GetCompletions_ExecContextSuggestsProcedures() + { + var result = SqlCompletionProvider.GetCompletions("EXEC Re", "EXEC Re".Length, CreateCatalog()); + + var suggestion = Assert.Single(result.Suggestions); + Assert.Equal("RefreshCustomerStats", suggestion.Label); + Assert.Equal(SqlCompletionSuggestionKind.Procedure, suggestion.Kind); + } + + [Fact] + public void GetCompletions_QualifiedAliasSuggestsAliasColumns() + { + var result = SqlCompletionProvider.GetCompletions("SELECT c. FROM Customers c", "SELECT c.".Length, CreateCatalog()); + + Assert.Contains(result.Suggestions, s => s.Label == "CustomerId"); + Assert.Contains(result.Suggestions, s => s.Label == "Name"); + } + + [Fact] + public void GetCompletions_AfterSetInUpdate_SuggestsColumnsForUpdateSource() + { + var result = SqlCompletionProvider.GetCompletions("UPDATE Customers SET ", "UPDATE Customers SET ".Length, CreateCatalog()); + + Assert.Contains(result.Suggestions, s => s.Label == "CustomerId"); + Assert.Contains(result.Suggestions, s => s.Label == "Name"); + Assert.Contains(result.Suggestions, s => s.Label == "Email"); + } + + [Fact] + public void GetCompletions_AfterWhereInUpdate_SuggestsColumnsForUpdateSource() + { + string sql = "UPDATE Customers SET Name = 'Alice' WHERE "; + + var result = SqlCompletionProvider.GetCompletions(sql, sql.Length, CreateCatalog()); + + Assert.Contains(result.Suggestions, s => s.Label == "CustomerId"); + Assert.Contains(result.Suggestions, s => s.Label == "Name"); + Assert.Contains(result.Suggestions, s => s.Label == "Email"); + } + + [Fact] + public void GetCompletions_ExactColumnMatch_DoesNotKeepPopupOpen() + { + string sql = "SELECT Name FROM Customers"; + + var result = SqlCompletionProvider.GetCompletions(sql, "SELECT Name".Length, CreateCatalog()); + + Assert.Empty(result.Suggestions); + } + + [Fact] + public void GetCompletions_ExactSourceMatch_DoesNotKeepPopupOpen() + { + string sql = "SELECT * FROM Customers"; + + var result = SqlCompletionProvider.GetCompletions(sql, sql.Length, CreateCatalog()); + + Assert.Empty(result.Suggestions); + } + + [Fact] + public void GetCompletions_ExplicitTriggerStillShowsSuggestionsForExactMatch() + { + string sql = "SELECT Name FROM Customers"; + + var result = SqlCompletionProvider.GetCompletions(sql, "SELECT Name".Length, CreateCatalog(), explicitTrigger: true); + + var suggestion = Assert.Single(result.Suggestions); + Assert.Equal("Name", suggestion.Label); + } + + [Fact] + public void GetCompletions_ExplicitTriggerWithEmptyPrefix_IncludesLimit() + { + var result = SqlCompletionProvider.GetCompletions("SELECT * FROM Customers ", "SELECT * FROM Customers ".Length, CreateCatalog(), explicitTrigger: true); + + Assert.Contains(result.Suggestions, s => s.Label == "LIMIT"); + Assert.Contains(result.Suggestions, s => s.Label == "OFFSET"); + } + + private static SqlCompletionCatalog CreateCatalog() + => new() + { + Sources = + [ + new SqlCompletionSource("Customers", SqlCompletionSourceKind.Table), + new SqlCompletionSource("Orders", SqlCompletionSourceKind.Table), + new SqlCompletionSource("CustomerSummary", SqlCompletionSourceKind.View), + new SqlCompletionSource("sys.tables", SqlCompletionSourceKind.SystemCatalog), + ], + ColumnsBySource = new Dictionary>(StringComparer.OrdinalIgnoreCase) + { + ["Customers"] = + [ + new SqlCompletionColumn("CustomerId", "INTEGER", "Customers"), + new SqlCompletionColumn("Name", "TEXT", "Customers"), + new SqlCompletionColumn("Email", "TEXT", "Customers"), + ], + ["Orders"] = + [ + new SqlCompletionColumn("OrderId", "INTEGER", "Orders"), + new SqlCompletionColumn("CustomerId", "INTEGER", "Orders"), + new SqlCompletionColumn("Total", "REAL", "Orders"), + ], + }, + Procedures = ["RefreshCustomerStats"], + }; +} diff --git a/www/architecture-reference.html b/www/architecture-reference.html new file mode 100644 index 00000000..09ceb4b6 --- /dev/null +++ b/www/architecture-reference.html @@ -0,0 +1,1140 @@ + + + + + + + + Architecture Source Reference — CSharpDB + + + + + + + + + + + + + + +
+
+ + +
+
Source reference. This page preserves the original long-form markdown content that previously lived at docs/architecture.md. For the shorter curated page, see Architecture.
+

CSharpDB Architecture

+

CSharpDB is a layered embedded database engine inspired by SQLite's architecture. + The core engine layers have clear responsibilities and mostly communicate with + adjacent layers. Above the engine, CSharpDB now exposes multiple consumer-facing + entry points, with CSharpDB.Client as the authoritative database API. It also + ships a reusable package-driven ETL pipeline runtime in CSharpDB.Pipelines + that is reused by the client, API, CLI, and Admin surfaces.

+

Layer Overview

+ 
┌────────────────────────────────────────────────────────────────────┐
+                │ Hosts / Applications                                               │
+                │ CSharpDB.Api   CSharpDB.Daemon   CSharpDB.Admin   CSharpDB.Cli     │
+                │ CSharpDB.Mcp                                                       │
+                ├────────────────────────────────────────────────────────────────────┤
+                │ Consumer Access Layer                                              │
+                │ CSharpDB.Client                     CSharpDB.Data                  │
+                │ ICSharpDbClient                     ADO.NET Provider               │
+                ├────────────────────────────────────────────────────────────────────┤
+                │ Data Movement Layer                                                │
+                │ CSharpDB.Pipelines                                                 │
+                │ Package Models / Validation / Orchestrator / Serialization         │
+                ├────────────────────────────────────────────────────────────────────┤
+                │ CSharpDB.Engine                                                    │
+                │ Database.OpenAsync / ExecuteAsync / Transactions / ReaderSession   │
+                ├────────────────────────────────────────────────────────────────────┤
+                │ CSharpDB.Execution                                                 │
+                │ QueryPlanner, Operators, ExpressionEvaluator                       │
+                ├───────────────────────────────┬────────────────────────────────────┤
+                │ CSharpDB.Sql                  │ CSharpDB.Storage                   │
+                │ Tokenizer, Parser, AST        │ Pager, B+Tree, WAL, RecordCodec    │
+                ├───────────────────────────────┴────────────────────────────────────┤
+                │ CSharpDB.Primitives                                                │
+                │ DbValue, DbType, Schema, ErrorCodes                                │
+                └────────────────────────────────────────────────────────────────────┘
+
+

Dependency graph:

+ 
Api     → Client
+                Daemon  → Client
+                Admin   → Client
+                Cli     → Client
+                Cli     → Engine              (local-only helpers)
+                Cli     → Sql
+                Cli     → Storage.Diagnostics
+                Cli     → Pipelines
+                Mcp     → Client
+                Data    → Client
+                Data    → Engine            (named shared-memory host + internal session types)
+                Client  → Engine
+                Client  → Pipelines
+                Client  → Sql
+                Client  → Storage.Diagnostics
+                Pipelines → Sql
+                Engine  → Execution → Sql
+                                    → Storage → Primitives
+                          Execution → Primitives
+                Engine  → Storage
+                Engine  → Sql
+                Engine  → Primitives
+
+
+

Layer 1: Primitives (CSharpDB.Primitives)

+

Shared types used by every other layer. No dependencies.

+ + + + + + + + + + + + + + + + + + + + + + + + + +
FilePurpose
DbType.csEnum: Null, Integer, Real, Text, Blob
DbValue.csDiscriminated union value type with comparison, equality, truthiness
Schema.csColumnDefinition, TableSchema, IndexSchema, TriggerSchema, and related metadata types
CSharpDbException.csException with ErrorCode enum (IoError, TableNotFound, SyntaxError, DuplicateKey, WalError, Busy, etc.)
+

DbValue

+

DbValue is a readonly struct that can hold any of the five database types. It uses a compact internal layout — a long for integers, a double for reals, and an object? reference for strings and byte arrays. The Type property indicates which field is active.

+

Key behaviors:

+
    +
  • Comparison: NULLs sort first. Integer and Real are cross-comparable via promotion to double. Text uses ordinal string comparison. Blob uses byte-by-byte comparison.
    • +
  • Truthiness: NULL and zero are falsy. Non-zero numbers, all strings, and all blobs are truthy. Used by WHERE clause evaluation.
    • +
+
+

Layer 2: Storage (CSharpDB.Storage)

+

The storage layer manages all on-disk data structures. It handles file I/O, page caching, crash-safe transactions via WAL, B+tree operations, secondary indexes, and record encoding.

+

File I/O

+ + + + + + + + + + + + + + + + + +
FilePurpose
IStorageDevice.csAbstract async interface: ReadAsync, WriteAsync, FlushAsync, SetLengthAsync
FileStorageDevice.csImplementation using System.IO.RandomAccess with FileOptions.Asynchronous
+

The storage device abstraction means the engine could be backed by any byte-addressable store (memory, network, encrypted file).

+

Page System

+ + + + + + + + + + + + + + + + + + + + + +
FilePurpose
PageConstants.csPage size (4096 bytes), file header layout, page types, WAL format constants
SlottedPage.csStructured access to slotted page layout (cells, pointers, free space)
Pager.csPage I/O, buffer pool, dirty tracking, page allocation/freelist, transaction lifecycle, WAL integration, snapshot readers
+

Database File Format

+

The database is a sequence of 4096-byte pages. Page 0 contains the file header:

+ 
Offset  Size  Field
+                ──────  ────  ─────
+                0       4     Magic bytes: "CSDB"
+                4       4     Format version (1)
+                8       4     Page size (4096)
+                12      4     Total page count
+                16      4     Schema catalog B+tree root page ID
+                20      4     Freelist head page ID (0 = empty)
+                24      4     Change counter
+                28      72    Reserved (zeroed)
+                100     ...   Page 0 content area (usable for B+tree data)
+
+

Slotted Page Layout

+

Each B+tree page uses a slotted page format:

+ 
┌───────────────────────────────────────────────────────────┐
+                │ Page Header (9 bytes)                                     │
+                │  [PageType:1] [CellCount:2] [ContentStart:2] [RightPtr:4] │
+                ├───────────────────────────────────────────────────────────┤
+                │ Cell Pointer Array (2 bytes each, grows forward →)        │
+                │  [ptr0] [ptr1] [ptr2] ...                                 │
+                ├───────────────────────────────────────────────────────────┤
+                │                    Free Space                             │
+                ├───────────────────────────────────────────────────────────┤
+                │ Cell Content Area (grows ← backward from page end)        │
+                │  ... [cell2] [cell1] [cell0]                              │
+                └───────────────────────────────────────────────────────────┘
+
+

The cell pointer array and cell content area grow toward each other. When they meet, the page is full and must be split.

+

Pager

+

The Pager is the central coordinator for page-level operations:

+
    +
  • Page cache: In-memory Dictionary<uint, byte[]> of loaded pages
    • +
  • Dirty tracking: HashSet<uint> of modified pages that need flushing
    • +
  • Allocation: Pages are allocated from a freelist (linked list of free page IDs) or by extending the page count
    • +
  • Transactions: Begin/Commit/Rollback lifecycle with WAL integration
    • +
  • Writer lock: SemaphoreSlim(1,1) ensures single-writer access
    • +
  • Snapshot readers: CreateSnapshotReader(snapshot) creates read-only pagers that see a frozen point-in-time view of the database
    • +
+

Write-Ahead Log (WAL)

+ + + + + + + + + + + + + + + + + +
FilePurpose
WriteAheadLog.csWAL file I/O — frame-based append, commit, rollback, checkpoint, crash recovery
WalIndex.csIn-memory index mapping pageId → WAL file offset, plus immutable snapshots
+

CSharpDB uses a Write-Ahead Log for crash recovery and concurrent reader support. Modified pages are appended to a .wal file during commit, while the main .db file retains old data until checkpoint.

+

WAL File Format

+ 
┌──────────────────────────────────────────────────────┐
+                │ WAL Header (32 bytes)                                │
+                │  [magic:"CWAL"] [version:4] [pageSize:4]             │
+                │  [dbPageCount:4] [salt1:4] [salt2:4]                 │
+                │  [checksumSeed:4] [reserved:4]                       │
+                ├──────────────────────────────────────────────────────┤
+                │ Frame 0 (4120 bytes)                                 │
+                │  [pageId:4] [dbPageCount:4] [salt1:4] [salt2:4]      │
+                │  [headerChecksum:4] [dataChecksum:4]                 │
+                │  [page data: 4096 bytes]                             │
+                ├──────────────────────────────────────────────────────┤
+                │ Frame 1 ...                                          │
+                ├──────────────────────────────────────────────────────┤
+                │ Frame N (commit frame: dbPageCount > 0)              │
+                └──────────────────────────────────────────────────────┘
+
+

Transaction Lifecycle (WAL Mode)

+ 
1. BEGIN TRANSACTION
+                   └── Acquire writer lock (SemaphoreSlim)
+                   └── Record WAL position
+                
+                2. MODIFY PAGES
+                   └── Track dirty pages in memory
+                   └── Pages are modified in the page cache
+                
+                3a. COMMIT
+                    └── Append all dirty pages as WAL frames
+                    └── Mark last frame as commit (dbPageCount > 0)
+                    └── Flush WAL according to configured durability policy (commit point)
+                    └── Update in-memory WAL index
+                    └── Release writer lock
+                    └── Auto-checkpoint if WAL exceeds threshold (default: 1000 frames)
+                
+                3b. ROLLBACK (or CRASH)
+                    └── Truncate WAL back to pre-transaction position
+                    └── Clear page cache
+                    └── Release writer lock
+
+

WAL Durability Modes

+

File-backed storage now exposes explicit WAL durability modes through + StorageEngineOptions.DurabilityMode:

+
    +
  • Durable: flushes managed buffers and forces the OS-backed WAL flush + before commit success is reported. This is the crash-safe default and is + analogous to SQLite WAL FULL.
    • +
  • Buffered: flushes managed buffers into the OS, but does not force an + OS-buffer flush on every commit. This is the higher-throughput mode and is + analogous to SQLite WAL NORMAL.
    • +
+

Internally, WriteAheadLog routes commit completion through an explicit + IWalFlushPolicy (DurableWalFlushPolicy or BufferedWalFlushPolicy) so the + durability tradeoff is visible at the storage boundary instead of being an + implicit side effect of file-stream behavior.

+

Durable commits also support grouped completion: when multiple writers reach the + flush boundary together, they can share one durable flush sequence. The pager's + commit wait is no longer held under the writer lock, which keeps single-writer + correctness intact while reducing unnecessary durable-commit contention.

+

Crash Recovery

+

On database open, if a .wal file exists, the WAL is scanned frame-by-frame. Committed transactions (those with a valid commit frame) are replayed into the WAL index, and a checkpoint copies all committed pages to the DB file.

+

Concurrent Readers

+

Readers acquire a snapshot — a frozen copy of the WAL index at a point in time. Each snapshot reader gets its own Pager instance that routes page reads through the snapshot. This means:

+
    +
  • Readers see a consistent point-in-time view
    • +
  • Writers do not block readers
    • +
  • Multiple readers can be active simultaneously
    • +
  • Checkpoint is skipped while readers are active (their snapshots reference WAL data)
    • +
+

B+Tree

+ + + + + + + + + + + + + + + + + +
FilePurpose
BTree.csB+tree keyed by long rowid — insert, delete, find, split
BTreeCursor.csForward-only cursor for sequential scans and seeks
+

Each table's data is stored in a B+tree where the key is an auto-generated rowid and the value is an encoded row. Secondary indexes also use B+trees.

+

Leaf page cell format:

+ 
[totalSize:varint] [key:8 bytes] [payload bytes...]
+
+

Interior page cell format:

+ 
[totalSize:varint] [leftChild:4 bytes] [key:8 bytes]
+
+

Interior pages also store a "rightmost child" pointer in the page header. Leaf pages are linked via a "next leaf" pointer for efficient sequential scans.

+

Operations:

+
    +
  • Insert: Descend to the correct leaf, insert the cell. If the leaf overflows, split it and propagate the split key upward. If the root splits, create a new root.
    • +
  • Delete: Descend to the leaf, remove the cell, rebalance underflowed pages by borrowing or merging when needed, and collapse an empty interior root back to its child.
    • +
  • Find: Descend from root to leaf following routing keys in interior pages.
    • +
  • Scan: The BTreeCursor starts at the leftmost leaf and follows next-leaf pointers.
    • +
+

Record Encoding

+ + + + + + + + + + + + + + + + + + + + + +
FilePurpose
RecordEncoder.csSerialize/deserialize DbValue[] rows to compact binary format
Varint.csLEB128 variable-length integer encoding
SchemaSerializer.csSerialize/deserialize TableSchema for the schema catalog
+

Row encoding format:

+ 
[columnCount:varint] [type1:1 byte] [type2:1 byte] ... [data1] [data2] ...
+
+

Where each data field is:

+
    +
  • Null: nothing (0 bytes)
    • +
  • Integer: varint-encoded long
    • +
  • Real: 8 bytes (IEEE 754 double)
    • +
  • Text: [length:varint] [UTF-8 bytes]
    • +
  • Blob: [length:varint] [raw bytes]
    • +
+

Schema Catalog

+ + + + + + + + + + + + + +
FilePurpose
SchemaCatalog.csIn-memory cache of table/index/view/trigger schemas, backed by dedicated B+trees
+

The schema catalog stores all database metadata in B+trees:

+
    +
  • Table schemas: table name, column definitions, root page ID
    • +
  • Index schemas: index name, table name, columns, uniqueness, root page ID
    • +
  • View definitions: view name → SQL text
    • +
  • Trigger definitions: trigger name, table, timing, event, body SQL
    • +
+

On database open, all schemas are loaded into in-memory dictionaries for fast lookups. When objects are created or dropped, both the in-memory cache and the on-disk B+trees are updated.

+
+

Layer 3: SQL Frontend (CSharpDB.Sql)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FilePurpose
TokenType.csEnum of all token types (keywords, operators, literals, punctuation)
Token.csToken struct: Type, Value (string), Position (int)
Tokenizer.csHand-rolled lexical scanner with keyword lookup table
Ast.csAST node classes for all statement and expression types
Parser.csRecursive descent parser with precedence climbing for expressions
+

Supported Statements

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CategoryStatements
DDLCREATE TABLE, DROP TABLE, ALTER TABLE (ADD/DROP COLUMN, RENAME TABLE/COLUMN)
DMLINSERT INTO, SELECT, UPDATE, DELETE
IndexesCREATE INDEX, DROP INDEX (with UNIQUE, IF NOT EXISTS/IF EXISTS, composite multi-column)
ViewsCREATE VIEW, DROP VIEW
TriggersCREATE TRIGGER, DROP TRIGGER (BEFORE/AFTER, INSERT/UPDATE/DELETE)
CTEsWITH ... AS (...) SELECT ...
Set operationsUNION, INTERSECT, EXCEPT (inside top-level queries, views, and CTE bodies)
SubqueriesScalar subqueries, IN (SELECT ...), EXISTS (SELECT ...), correlated evaluation in WHERE, non-aggregate projection, UPDATE/DELETE expressions
StatisticsANALYZE [table] — refreshes sys.table_stats and sys.column_stats
IdentityINTEGER PRIMARY KEY IDENTITY — auto-increment columns with persisted high-water mark
DistinctSELECT DISTINCT, DISTINCT inside aggregates
+

Parsing Pipeline

+ 
SQL string → Tokenizer → Token[] → Parser → AST (Statement tree)
+
+

The tokenizer scans the input character by character, recognizing keywords (case-insensitive), identifiers, numeric literals (integer and real), string literals (single-quoted with '' escaping), operators, and punctuation.

+

The parser is a recursive descent parser. Each SQL statement type has its own parsing method. Expression parsing uses precedence climbing to correctly handle operator precedence:

+ 
Precedence (low to high):
+                  OR
+                  AND
+                  NOT (unary)
+                  =, <>, <, >, <=, >=, LIKE, IN, BETWEEN, IS NULL
+                  +, -
+                  *, /
+                  - (unary)
+
+
+

Layer 4: Execution (CSharpDB.Execution)

+ + + + + + + + + + + + + + + + + + + + + + + + + +
FilePurpose
IOperator.csIterator interface: OpenAsync, MoveNextAsync, Current
Operators.csPhysical operators: TableScan, IndexScan, Filter, Projection, Sort, Limit, Aggregate, Join, etc.
ExpressionEvaluator.csEvaluates expression AST against a row (including LIKE, IN, BETWEEN, IS NULL, aggregates)
QueryPlanner.csConverts AST statements into executable operator trees or DML/DDL actions
+

Iterator Model

+

Query execution follows the Volcano/iterator model. Each operator implements IOperator:

+ 
public interface IOperator : IAsyncDisposable
+                {
+                    ColumnDefinition[] OutputSchema { get; }
+                    ValueTask OpenAsync(CancellationToken ct = default);
+                    ValueTask<bool> MoveNextAsync(CancellationToken ct = default);
+                    DbValue[] Current { get; }
+                }
+
+

Operators form a tree. The root operator pulls rows upward by calling MoveNextAsync on its child, which in turn calls its child, and so on down to the leaf scan operator.

+

Operator Catalog

+

Scan and Lookup Operators

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorPurpose
TableScanOperatorFull table scan via BTreeCursor — batch-capable
IndexScanOperatorIndex-based lookup with base-row fetch — batch-capable
IndexOrderedScanOperatorOrdered index range scan — batch-capable
UniqueIndexLookupOperatorSingle-row unique index probe
PrimaryKeyLookupOperatorDirect rowid B+tree lookup fast path
PrimaryKeyProjectionLookupOperatorPK lookup with projection pushdown
UniqueIndexProjectionLookupOperatorUnique index lookup with projection pushdown
HashedIndexProjectionLookupOperatorHashed index lookup with projection pushdown
IndexScanProjectionOperatorIndex scan with index-only projection
IndexOrderedProjectionScanOperatorOrdered index scan with index-only projection
+

Filter and Projection Operators

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorPurpose
FilterOperatorApplies a WHERE predicate — batch-capable
ProjectionOperatorSelects/reorders columns, evaluates expressions — batch-capable
FilterProjectionOperatorFused filter + projection for lower materialization
CompactTableScanProjectionOperatorCompact scan with fused filter/projection over encoded rows
CompactPayloadProjectionOperatorCompact projection over encoded index/table payloads
+

Join Operators

+ + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorPurpose
HashJoinOperatorHash-based join with projection pushdown
IndexNestedLoopJoinOperatorIndex-probed nested loop join
HashedIndexNestedLoopJoinOperatorHashed index probe nested loop join
NestedLoopJoinOperatorGeneric INNER, LEFT, RIGHT, and CROSS JOINs
+

Aggregate Operators

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorPurpose
HashAggregateOperatorGROUP BY with hash-based grouping
ScalarAggregateOperatorSingle-group aggregate (no GROUP BY)
IndexKeyAggregateOperatorIndex-backed single-key aggregate fast path
IndexGroupedAggregateOperatorIndex-backed grouped aggregate
CompositeIndexGroupedAggregateOperatorComposite index grouped aggregate
TableKeyAggregateOperatorTable-key aggregate fast path
ScalarAggregateLookupOperatorScalar aggregate over lookup result
ScalarAggregateTableOperatorScalar aggregate over full table scan
FilteredScalarAggregateTableOperatorFiltered scalar aggregate fast path
CountStarTableOperatorOptimized COUNT(*) over table metadata
+

Sort, Distinct, and Limit Operators

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorPurpose
SortOperatorMaterializes all input, sorts by ORDER BY — batch-capable
TopNSortOperatorHeap-based ORDER BY + LIMIT without full materialization
DistinctOperatorHash-based SELECT DISTINCT — batch-capable
OffsetOperatorSkips N rows — batch-capable
LimitOperatorCaps output at N rows — batch-capable
MaterializedOperatorPre-materialized row set (used for CTEs and subqueries)
+

Query Planning

+

For SELECT, the planner builds an operator tree:

+ 
TableScan/IndexScan → [Filter] → [Join] → [Aggregate] → [Having] → [Sort] → [Projection] → [Limit]
+
+

The planner includes index selection with multiple strategies: equality lookups on indexed columns, ordered range scans on integer indexes, composite index matching, covering-index projection pushdown, and statistics-guided non-unique lookup selection via sys.column_stats.

+

For DML (INSERT, UPDATE, DELETE) and DDL (CREATE/DROP/ALTER TABLE, CREATE/DROP INDEX, CREATE/DROP VIEW, CREATE/DROP TRIGGER), the planner executes the operation directly against the B+tree and schema catalog, returning a row-count result.

+

Triggers are fired automatically during INSERT, UPDATE, and DELETE operations. The planner checks for BEFORE and AFTER triggers on the affected table and executes their body SQL statements. A recursion guard prevents infinite trigger chains (max depth: 16).

+

Views are expanded inline during query planning — a reference to a view in a FROM clause is replaced with the view's SQL definition, parsed and planned recursively.

+

CTEs (WITH clause) are materialized eagerly — the CTE query is executed first and its results are stored in memory, then referenced by the main query.

+

Expression Evaluator

+

The ExpressionEvaluator is a static class that recursively evaluates an Expression AST node against a current row. It handles:

+
    +
  • Column references — look up by column name (or qualified table.column) in the schema
    • +
  • Literals — integer, real, text, null
    • +
  • Binary operators — arithmetic (+, -, *, /), comparison (=, <>, <, >, <=, >=), logical (AND, OR)
    • +
  • Unary operators — NOT, negation
    • +
  • LIKE — pattern matching with % and _ wildcards, optional ESCAPE character
    • +
  • IN — membership test against a list of values or IN (SELECT ...)
    • +
  • BETWEEN — range check (inclusive)
    • +
  • IS NULL / IS NOT NULL — null testing
    • +
  • Aggregate functions — COUNT, SUM, AVG, MIN, MAX (with DISTINCT support)
    • +
  • Scalar functions — TEXT(expr) for filter-friendly text coercion
    • +
  • Scalar subqueries — single-value subquery evaluation, including correlated cases
    • +
  • EXISTS (SELECT ...) — existence test subquery evaluation
    • +
+
+

Layer 5: Engine (CSharpDB.Engine)

+ + + + + + + + + + + + + + + + + +
FilePurpose
Database.csTop-level API: file-backed, in-memory, and hybrid open modes; execute SQL; manage transactions, checkpoints, and reader sessions
Collection.csTyped document collection API backed by storage-engine B+trees
+

The Database class ties all layers together:

+
    +
  1. Open: Opens the database in file-backed, fully in-memory, or hybrid lazy-resident mode. Supports opt-in memory-mapped main-file reads, storage tuning presets (UseLookupOptimizedPreset, UseWriteOptimizedPreset), bounded WAL read caching, and background sliced auto-checkpointing. Runs crash recovery if a WAL file exists and loads the schema catalog.
    2. +
  2. ExecuteAsync: Parses SQL → dispatches to QueryPlanner → returns QueryResult
    3. +
  3. Auto-commit: Non-SELECT statements automatically begin and commit a transaction if none is active
    4. +
  4. Explicit transactions: BeginTransactionAsync / CommitAsync / RollbackAsync for the legacy single-handle transaction shape, plus BeginWriteTransactionAsync(...) / RunWriteTransactionAsync(...) for isolated multi-writer work with conflict detection and retry
    5. +
  5. CheckpointAsync: Manually triggers a WAL checkpoint (copies committed WAL pages to DB file)
    6. +
  6. CreateReaderSession: Returns an independent ReaderSession that sees a snapshot of the database at the current point in time. Multiple reader sessions can coexist with an active writer.
    7. +
  7. Collections: GetCollectionAsync<T> exposes the typed document path beside SQL with binary direct-payload storage, path-based indexing (EnsureIndexAsync, FindByPathAsync, FindByPathRangeAsync), and direct binary hydration
    8. +
  8. Dispose: Rolls back any uncommitted transaction, checkpoints, deletes WAL file, and closes the pager
    9. +
+
+

Layer 6: Pipelines (CSharpDB.Pipelines)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FilePurpose
Models/PipelinePackageDefinition.csPackage model for sources, transforms, destinations, execution options, and incremental settings
Models/PipelineRuntimeModels.csRun request/result, metrics, checkpoints, rejects, and execution context
Validation/PipelinePackageValidator.csValidates package completeness and supported configuration combinations
Runtime/PipelineOrchestrator.csExecutes validate, dry-run, run, and resume flows
Runtime/BuiltIns/*Built-in CSV/JSON file sources and destinations plus built-in transforms
Serialization/PipelinePackageSerializer.csJSON persistence for package files and stored package payloads
+

CSharpDB.Pipelines is the reusable ETL pipeline runtime. It is intentionally + separate from the storage engine so package validation, orchestration, and + connector/transform logic can be reused from local and remote hosts without + mixing ETL concerns into the SQL execution pipeline.

+

Current responsibilities:

+
    +
  1. Package model: defines JSON-serializable pipeline packages with metadata, + source, transforms, destination, execution options, and optional incremental + settings.
    2. +
  2. Validation: validates package completeness and returns structured + validation errors before runtime execution starts.
    3. +
  3. Execution modes: supports Validate, DryRun, Run, and Resume.
    4. +
  4. Batch orchestration: opens the source, reads row batches, applies the + ordered transform chain, writes destination batches, and updates run metrics.
    5. +
  5. Checkpoint and run logging contracts: persists pipeline progress and run + status through IPipelineCheckpointStore and IPipelineRunLogger.
    6. +
  6. Built-in file connectors: includes CSV/JSON file sources and CSV/JSON + file destinations in the core runtime.
    7. +
+

The pipeline runtime is package- and batch-oriented, not a general DAG + scheduler. The current shipping model is a linear source -> transforms -> + destination flow with resumable batch checkpoints and reject tracking.

+
+

Layer 7: Unified Client (CSharpDB.Client)

+

CSharpDB.Client is the authoritative database API for CSharpDB consumers.

+

It owns the public client contract and transport selection boundary used by the + CLI, Web API, Admin dashboard, and future external consumers. Transport details + stay behind this layer.

+

Key pieces:

+
    +
  • ICSharpDbClient — transport-agnostic database contract
    • +
  • CSharpDbClientOptions — endpoint / data source / connection string options
    • +
  • CSharpDbTransport — public transport selector
    • +
  • AddCSharpDbClient(...) — DI registration helper
    • +
+

Current direction:

+
    +
  • Direct transport is implemented today and is backed by CSharpDB.Engine
    • +
  • HTTP transport is implemented and targets CSharpDB.Api
    • +
  • gRPC transport is implemented and targets CSharpDB.Daemon
    • +
  • Named Pipes is part of the public transport model but is not implemented yet
    • +
  • The client does not depend on CSharpDB.Data
    • +
  • New database-facing functionality should be added here first
    • +
+

Current surface includes:

+
    +
  • database info and metadata
    • +
  • table schemas, browse, CRUD, and table/column DDL
    • +
  • indexes, views, and triggers
    • +
  • saved queries
    • +
  • procedures and procedure execution
    • +
  • SQL execution
    • +
  • client-managed transactions
    • +
  • document collections
    • +
  • pipeline catalog, package storage, pipeline execution, resume, checkpoints, and rejects
    • +
  • checkpoint and storage diagnostics
    • +
  • backup and restore (BackupAsync, RestoreAsync)
    • +
  • maintenance (reindex, vacuum, maintenance report)
    • +
+

Implementation dependencies:

+
    +
  • CSharpDB.Engine
    • +
  • CSharpDB.Pipelines
    • +
  • CSharpDB.Sql
    • +
  • CSharpDB.Storage.Diagnostics
    • +
+

This means the current direct client is a high-level engine-backed API, not an + ADO.NET wrapper.

+
+

Pipeline Integration Through The Client

+

Pipeline management and execution are layered on top of ICSharpDbClient:

+
    +
  • CSharpDbPipelineRunner wraps the reusable PipelineOrchestrator
    • +
  • CSharpDbPipelineComponentFactory adds CSharpDB-backed table and SQL-query + connectors on top of the runtime's built-in file connectors
    • +
  • CSharpDbPipelineCatalogClient persists packages, revisions, runs, + checkpoints, and rejects in catalog tables such as _etl_pipelines, + _etl_pipeline_versions, _etl_runs, _etl_checkpoints, and _etl_rejects
    • +
+

This keeps ETL transport-agnostic: the same package/run/catalog flow works + through direct, HTTP, and gRPC clients because the persistence and execution + path are built on the same client contract.

+
+

Layer 8: ADO.NET Provider (CSharpDB.Data)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FilePurpose
CSharpDbConnection.csDbConnection implementation — open, close, connection string parsing
CSharpDbCommand.csDbCommand implementation — parameterized SQL execution
CSharpDbDataReader.csDbDataReader implementation — forward-only row iteration with typed accessors
CSharpDbParameter.csDbParameter / DbParameterCollection for parameterized queries
CSharpDbTransaction.csDbTransaction for explicit transaction control
CSharpDbFactory.csDbProviderFactory for ADO.NET provider registration
SqlParameterBinder.csBinds @param placeholders in SQL to parameter values
TypeMapper.csMaps between CSharpDB types and .NET CLR types
+

The ADO.NET provider allows CSharpDB to be used with the standard System.Data.Common APIs, making it compatible with ORMs and existing .NET data access code:

+ 
await using var conn = new CSharpDbConnection("Data Source=myapp.db");
+                await conn.OpenAsync();
+                
+                using var cmd = conn.CreateCommand();
+                cmd.CommandText = "SELECT * FROM users WHERE age > @age";
+                cmd.Parameters.AddWithValue("@age", 25);
+                
+                await using var reader = await cmd.ExecuteReaderAsync();
+                while (await reader.ReadAsync())
+                {
+                    Console.WriteLine(reader.GetString(1));
+                }
+
+

Today the provider sits mostly above CSharpDB.Client, not beside it:

+
    +
  • ordinary file-backed direct connections route through CSharpDB.Client
    • +
  • private :memory: connections route through CSharpDB.Client
    • +
  • daemon-backed Transport=Grpc;Endpoint=... connections route through CSharpDB.Client
    • +
  • named shared :memory:name stays as the one internal engine-assisted exception for now because that host is process-local state inside the provider
    • +
+

That means ADO.NET is now primarily a provider-shaped facade over the same + authoritative client contract used by the other host surfaces.

+
+

Layer 9: Remote Hosts (CSharpDB.Api + CSharpDB.Daemon)

+

The remote host split is intentional today:

+
    +
  • CSharpDB.Api is the REST/HTTP host
    • +
  • CSharpDB.Daemon is the gRPC host
    • +
+

Both inject ICSharpDbClient directly and stay above the authoritative + CSharpDB.Client contract instead of exposing engine internals.

+

REST API (CSharpDB.Api)

+

The REST API exposes the full database feature set over HTTP using ASP.NET Core Minimal APIs. It enables cross-language interoperability — any language with an HTTP client can work with CSharpDB.

+

Components:

+
    +
  • Endpoints — organized by resource (tables, rows, indexes, views, triggers, procedures, SQL, pipelines, info, inspection)
    • +
  • DTOs — Request/response records for type-safe serialization
    • +
  • JSON helpers — Coerce System.Text.Json JsonElement values to CLR primitives for the client
    • +
  • Exception middleware — Maps CSharpDbException error codes to HTTP status codes (404, 409, 422, etc.)
    • +
  • OpenAPI + Scalar — Auto-generated API spec with interactive documentation at /scalar
    • +
+

The API now injects ICSharpDbClient directly. It does not depend on + CSharpDB.Data or engine internals directly.

+

gRPC Host (CSharpDB.Daemon)

+

The daemon exposes explicit generated gRPC methods over the same client-facing + contract. It is a thin transport host, not a separate database engine.

+

Components:

+
    +
  • Generated protobuf contract — csharpdb_rpc.proto in CSharpDB.Client
    • +
  • GrpcTransportClient — remote client implementation in CSharpDB.Client
    • +
  • CSharpDbRpcService — gRPC method host in CSharpDB.Daemon
    • +
  • Startup validation — resolves ICSharpDbClient and validates the configured database during host startup
    • +
+

The current daemon default host shape is:

+
    +
  • direct transport internally
    • +
  • hybrid incremental-durable open mode
    • +
  • ImplicitInsertExecutionMode = ConcurrentWriteTransactions
    • +
  • UseWriteOptimizedPreset = true
    • +
  • optional hot-table / hot-collection preload hints through CSharpDB:HostDatabase
    • +
+

See the REST API Reference for HTTP details and the Daemon README for the gRPC host design.

+
+

Layer 10: Admin Dashboard (CSharpDB.Admin)

+

A Blazor Server application that provides a web-based UI for database administration. Features:

+
    +
  • Tab-based interface for browsing tables, views, indexes, and triggers
    • +
  • Paginated data grid with column headers
    • +
  • SQL execution panel
    • +
  • Procedure editing and execution
    • +
  • Pipeline designer and execution workflows
    • +
  • Storage inspection
    • +
  • Schema introspection (columns, types, constraints)
    • +
+

The Admin dashboard now injects ICSharpDbClient directly. It uses an + admin-local change notification service to refresh UI state after mutations. + Its pipeline UI works with package JSON plus a designer surface for source, + transform, destination, and execution-option editing, then routes execution and + catalog operations through the client-backed pipeline services.

+
+

Layer 11: CLI And MCP Hosts

+

Two additional host applications sit above the consumer access layer:

+
    +
  • CSharpDB.Cli — the interactive shell and local tooling entrypoint. It + now routes normal database access through CSharpDB.Client, while still + keeping a few local-only direct helpers for engine- and diagnostics-specific + features. It also exposes pipeline commands for validate, dry-run, run, + resume, import/export, and catalog inspection.
    • +
  • CSharpDB.Mcp — the MCP server host. It resolves ICSharpDbClient + directly and shares the same client configuration model as the other hosts.
    • +
+
+

End-to-End: Life of a Query

+

Here's what happens when you call db.ExecuteAsync("SELECT name FROM users WHERE age > 25 ORDER BY name"):

+ 
1. Parser.Parse(sql)
+                   ├── Tokenizer: "SELECT" "name" "FROM" "users" "WHERE" "age" ">" "25" "ORDER" "BY" "name"
+                   └── Parser: SelectStatement { Columns=[name], Table=users, Where=age>25, OrderBy=[name ASC] }
+                
+                2. QueryPlanner.ExecuteSelect(stmt)
+                   ├── Resolve "users" → TableSchema (from SchemaCatalog)
+                   ├── Check for usable index on WHERE columns
+                   ├── Build: TableScanOperator(users_btree) or IndexScanOperator if applicable
+                   ├── Wrap:  FilterOperator(scan, "age > 25")
+                   ├── Wrap:  SortOperator(filter, [name ASC])
+                   └── Wrap:  ProjectionOperator(sort, [name])
+                   └── Return: QueryResult(projectionOp)
+                
+                3. User calls result.GetRowsAsync()
+                   └── Opens operator chain top-down
+                       └── ProjectionOp.MoveNextAsync()
+                           └── SortOp.MoveNextAsync()  [materializes all matching rows, sorts]
+                               └── FilterOp.MoveNextAsync()  [skips rows where age <= 25]
+                                   └── TableScanOp.MoveNextAsync()
+                                       └── BTreeCursor.MoveNextAsync()
+                                           └── Pager.GetPageAsync(leafPageId)
+                                               ├── Check page cache
+                                               ├── Check WAL index for latest version
+                                               └── Fall through to FileStorageDevice.ReadAsync(offset)
+
+

Each row flows upward through the operator chain, transformed at each stage, until it reaches the caller.

+
+

See Also

+ +
+
+
+ + + + + diff --git a/www/architecture.html b/www/architecture.html index 64942f42..ea86b904 100644 --- a/www/architecture.html +++ b/www/architecture.html @@ -45,6 +45,9 @@

Architecture

CSharpDB is built in clean layers, each with a single responsibility. From client access down to file I/O.

+
+ Need the full source guide? The original long-form markdown version is preserved as Architecture Source Reference. +

Layered Architecture

diff --git a/www/blog/csharpdb-vs-sqlite-benchmark-guide.html b/www/blog/csharpdb-vs-sqlite-benchmark-guide.html index cb3c3873..ea2a699b 100644 --- a/www/blog/csharpdb-vs-sqlite-benchmark-guide.html +++ b/www/blog/csharpdb-vs-sqlite-benchmark-guide.html @@ -113,6 +113,9 @@

CSharpDB vs SQLite: A Benchmark-Driven Guide

+
+ Need the full long-form article? The original markdown version is preserved as CSharpDB vs SQLite Benchmarking Source Reference. +

Both CSharpDB and SQLite are excellent embedded databases for .NET, and "which one is faster?" is the wrong question. The right question is: for each workload, which API on which engine gets you the best number? Sometimes that's SQLite through ADO.NET. Sometimes it's CSharpDB through the direct engine API. Sometimes the gap between the two engines is smaller than the gap between a good API choice and a bad one on either side.

This post walks through four paired benchmarks on .NET 10 — bulk inserts, point lookups, concurrent writers, and EF Core — and shows the compilable code that produced each number. Everything in this post runs end-to-end with dotnet run -c Release.

diff --git a/www/blog/csharpdb-vs-sqlite-benchmarking-reference.html b/www/blog/csharpdb-vs-sqlite-benchmarking-reference.html new file mode 100644 index 00000000..f10d7ad5 --- /dev/null +++ b/www/blog/csharpdb-vs-sqlite-benchmarking-reference.html @@ -0,0 +1,516 @@ + + + + + + + + CSharpDB vs SQLite Benchmarking Source Reference — CSharpDB + + + + + + + + + + + + + + +
+
+ + +
+
Source reference. This page preserves the original long-form markdown content that previously lived at docs/query-and-durable-write-performance/csharpdb-vs-sqlite-benchmarking-blog.md. For the shorter curated page, see CSharpDB vs SQLite: A Benchmark-Driven Guide.
+

CSharpDB Versus SQLite: How We Compare Them Fairly, and How To Get the Most Out of CSharpDB

+

If you want to compare an embedded database against SQLite, it is very easy to + publish a misleading chart without meaning to.

+

The trap is always the same: one side gets measured at the engine layer, the + other side gets measured through a provider or ORM, durability settings do not + match, batching rules differ, and the final table looks precise while answering + the wrong question.

+

This post is the practical version of how we compare CSharpDB and SQLite in + this repository. It also doubles as a guide for getting the best insert + performance out of CSharpDB when you are using it in a real application.

+

The short version is:

+
    +
  • Compare engine to engine, provider to provider, and ORM to ORM.
    • +
  • Keep durability contracts aligned.
    • +
  • Reuse prepared work on both sides.
    • +
  • Treat single-writer bulk ingest and concurrent writers as different + workloads.
    • +
  • Use the CSharpDB surfaces that avoid per-row SQL and planner overhead when + you care about write throughput.
    • +
+

Why Fair Comparisons Are Hard

+

The phrase "CSharpDB versus SQLite" sounds like one benchmark, but in practice + it is at least four different comparisons:

+
    +
  1. CSharpDB engine API versus SQLite engine API.
    2. +
  2. CSharpDB ADO.NET provider versus SQLite ADO.NET provider.
    3. +
  3. CSharpDB EF Core provider versus SQLite EF Core provider.
    4. +
  4. Single-writer durable bulk ingest versus concurrent writer contention.
    5. +
+

Those are not interchangeable.

+

If one result uses the raw engine on one side and ADO.NET on the other, the + chart is already mixing surfaces. If one result uses file-backed durable WAL + and the other uses shared memory, it is no longer a storage-engine comparison. + If one side prepares a command once and reuses it while the other reparses SQL + for every row, the result mostly measures statement setup rather than insert + throughput.

+

That is why the CSharpDB benchmark suite splits these questions on purpose.

+

The Comparison Rules We Use

+

The repo follows a few rules consistently.

+

Rule 1: Match the Layer

+

We keep these pairings separate:

+
    +
  • Engine versus engine.
    • +
  • ADO.NET versus ADO.NET.
    • +
  • EF Core versus EF Core.
    • +
+

That means:

+
    +
  • If we compare CSharpDB directly through Database and InsertBatch, we + compare it against SQLite's native C API rather than against + Microsoft.Data.Sqlite.
    • +
  • If we compare CSharpDB.Data, we compare it against + Microsoft.Data.Sqlite.
    • +
  • If we compare the EF Core provider, we compare it against EF Core on SQLite.
    • +
+

Rule 2: Match the Durability Contract

+

For the primary durable write comparison, both sides are file-backed and + durable.

+

For the matched SQLite baseline in + SqliteComparisonBenchmark.cs, + SQLite uses:

+
    +
  • journal_mode=WAL
    • +
  • synchronous=FULL
    • +
  • explicit transaction batching
    • +
  • prepared statement reuse
    • +
+

For the corresponding CSharpDB bulk path in + DurableSqlBatchingBenchmark.cs, + the comparison uses the engine's durable file-backed path with the write-heavy + storage preset and the same logical batch shape.

+

Rule 3: Match the Statement Shape

+

SQLite absolutely does support the classic "prepare once, bind many times" + workflow. In the native API that is sqlite3_prepare_v2, followed by repeated + binds and sqlite3_step / reset cycles. In ADO.NET the equivalent is + DbCommand.Prepare() plus reusing parameters.

+

CSharpDB has two comparable fast paths, depending on the layer:

+
    +
  • At the engine layer, use Database.PrepareInsertBatch(...).
    • +
  • At the ADO.NET layer, use DbCommand.Prepare() through CSharpDB.Data.
    • +
+

If you compare SQLite with prepared reuse against CSharpDB with one fresh SQL + string per row, you are not comparing the engines fairly.

+

Rule 4: Keep Single-Writer and Multi-Writer Results Separate

+

Single-writer durable ingest answers one question:

+

How fast can each engine append committed rows when batching and durability are + held constant?

+

Concurrent writers answer a different question:

+

How does each engine behave when several local writers are inserting at the + same time against the same database?

+

Those are different workloads with different limits. They should not be merged + into one headline number.

+

What the Current Durable Bulk Numbers Say

+

The cleanest apples-to-apples insert comparison in this repo is still the + single-writer durable four-column bulk row:

+
    +
  • schema: + id INTEGER PRIMARY KEY, value INTEGER, text_col TEXT, category TEXT
    • +
  • monotonic primary keys
    • +
  • explicit transaction batching
    • +
  • file-backed durable mode
    • +
  • median-of-3 reruns on the same runner
    • +
+

As of April 21, 2026 (promoted from the release-core run with + PASS=185, WARN=0, SKIP=0, FAIL=0), the latest matched rerun on this machine + produced:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ScenarioRows/secP50P99
CSharpDB InsertBatch B1000204,0284.1034 ms9.5599 ms
SQLite prepared bulk B1000192,0594.7479 ms18.1078 ms
CSharpDB InsertBatch B10000798,2548.7019 ms119.0664 ms
SQLite prepared bulk B10000539,56216.6275 ms43.3034 ms
+

That means, on this runner:

+
    +
  • CSharpDB is about 106.2% of SQLite at B1000.
    • +
  • CSharpDB is about 147.9% of SQLite at B10000.
    • +
+

Source artifacts:

+
    +
  • durable-sql-batching-20260421-214227-median-of-3.csv
    • +
  • sqlite-compare-20260421-222824-median-of-3.csv
    • +
  • Full scorecard: tests/CSharpDB.Benchmarks/SQLITE_COMPARISON.md
    • +
+

The important part is not just that CSharpDB is ahead on this matched row. The + important part is what that row actually proves:

+
    +
  • It proves the primary durable monotonic bulk path is now competitive.
    • +
  • It does not prove every insert shape is faster.
    • +
  • It does not erase the cost of secondary indexes.
    • +
  • It does not erase the random-key locality cliff.
    • +
  • It does not answer hot right-edge multi-writer contention.
    • +
+

That distinction matters if you want the public write-up to stay honest.

+

The Three Real Benchmark Surfaces

+

The benchmark suite now lets us talk about three comparison layers clearly.

+

1. Direct Engine Versus SQLite C API

+

This comparison lives in + ConcurrentSqliteCApiComparisonBenchmark.cs.

+

On the CSharpDB side:

+
    +
  • one shared Database instance is opened for the scenario
    • +
  • one writer task is created per logical writer
    • +
  • each writer loops for the measurement window
    • +
  • each loop performs one insert commit
    • +
  • the surface is either raw Database.ExecuteAsync(sql) or a reused + one-row InsertBatch
    • +
  • concurrent insert routing can opt into + ImplicitInsertExecutionMode.ConcurrentWriteTransactions
    • +
+

On the SQLite side:

+
    +
  • one native SQLite connection handle is opened per writer
    • +
  • one prepared native insert statement is created per writer
    • +
  • each loop rebinds values, calls sqlite3_step, then resets the statement
    • +
  • SQLite runs with WAL, FULL, and a busy_timeout
    • +
+

This is the right layer for answering engine-versus-engine concurrency + questions.

+

2. ADO.NET Versus ADO.NET

+

This comparison lives in + ConcurrentAdoNetComparisonBenchmark.cs + and + StrictInsertComparisonBenchmark.cs.

+

The provider rules are intentionally symmetrical:

+
    +
  • one connection per writer
    • +
  • one prepared command per writer
    • +
  • one ExecuteNonQueryAsync() call per insert
    • +
+

The strict single-writer comparison also covers:

+
    +
  • raw SQL single inserts
    • +
  • prepared single inserts
    • +
  • raw SQL batched inserts
    • +
  • prepared batched inserts
    • +
  • file-backed CSharpDB ADO.NET opened with + Storage Preset=WriteOptimized and Embedded Open Mode=Direct
    • +
  • file-backed SQLite ADO.NET with journal_mode=WAL and synchronous=FULL
    • +
+

This is the right layer for answering provider overhead questions. It is not + the right layer for making storage-engine claims unless the durability contract + also matches.

+

3. EF Core Versus EF Core

+

This comparison lives in + EfCoreComparisonBenchmark.cs.

+

Today it covers:

+
    +
  • --efcore-compare: single-row and 100-row SaveChangesAsync() with one + open connection held for the timed run
    • +
  • --efcore-compare-hybrid-shared-connection: the same insert shapes with + one externally-owned open connection reused across short-lived DbContext + instances
    • +
  • --efcore-compare-auto-open-close: the same insert shapes with EF-managed + auto-open/close left in the measurement
    • +
  • CSharpDB EF Core provider versus SQLite EF Core provider
    • +
  • CSharpDB EF Core configured with UseStoragePreset(WriteOptimized) and + UseEmbeddedOpenMode(Direct)
    • +
+

It does not currently provide a dedicated concurrent EF Core benchmark. So if + you are writing a blog post, the fair statement is:

+

We have an EF Core comparison surface, but the concurrent comparison story is + currently strongest at the engine and ADO.NET layers.

+

How Concurrent Writer Tests Differ Between the Two Databases

+

This is usually where comparison posts become hand-wavy, so it is worth being + explicit.

+

For CSharpDB concurrent engine tests:

+
    +
  • writers share one in-process engine
    • +
  • the engine decides whether inserts stay serialized or route through isolated + write transactions
    • +
  • disjoint explicit-key workloads can benefit from the concurrent insert path
    • +
  • hot right-edge workloads still behave very differently from disjoint-key + workloads
    • +
+

For SQLite concurrent engine tests:

+
    +
  • each writer has its own native handle to the same database file
    • +
  • every writer still competes inside SQLite's WAL write rules
    • +
  • prepared statements are reused, but the engine remains effectively + single-writer at the durable file boundary
    • +
+

So the concurrency comparison is not "who has more threads." It is "how does + each engine's write architecture behave when several local writers are active."

+

How To Get the Best Performance Out of CSharpDB

+

This is the part that matters if you are not just comparing benchmarks but also + trying to ship a fast application.

+

Start With the Engine Fast Path

+

If you are using the engine directly, the highest-value insert surface is + PrepareInsertBatch(...), not one dynamically built SQL string per row.

+ 
using CSharpDB.Engine;
+                using CSharpDB.Primitives;
+                
+                var options = new DatabaseOptions()
+                    .ConfigureStorageEngine(builder => builder.UseWriteOptimizedPreset());
+                
+                await using var db = await Database.OpenAsync("ingest.db", options);
+                
+                await db.ExecuteAsync("""
+                    CREATE TABLE IF NOT EXISTS bench (
+                        id INTEGER PRIMARY KEY,
+                        value INTEGER,
+                        text_col TEXT,
+                        category TEXT
+                    )
+                    """);
+                
+                var batch = db.PrepareInsertBatch("bench", initialCapacity: 1000);
+                
+                await db.BeginTransactionAsync();
+                try
+                {
+                    for (int block = 0; block < 100; block++)
+                    {
+                        for (int i = 0; i < 1000; i++)
+                        {
+                            int id = (block * 1000) + i;
+                            batch.AddRow(
+                                DbValue.FromInteger(id),
+                                DbValue.FromInteger(id),
+                                DbValue.FromText("durable_batch"),
+                                DbValue.FromText("Alpha"));
+                        }
+                
+                        await batch.ExecuteAsync();
+                    }
+                
+                    await db.CommitAsync();
+                }
+                catch
+                {
+                    await db.RollbackAsync();
+                    throw;
+                }
+
+

The reason this matters is straightforward: it keeps the workload on the engine + bulk path instead of paying repeated SQL parse and planner cost for every row.

+

Batch Commits Aggressively Before You Reach for Exotic Knobs

+

The first major durable-write lever is still rows per commit.

+

The practical starting guidance from the current benchmark matrix is:

+
    +
  • Start at 1000 rows per commit.
    • +
  • Measure 10000 only for dedicated ingest jobs that can tolerate larger + commit latency and larger WAL bursts.
    • +
+

Single-row durable auto-commit inserts are valid for correctness tests and for + latency-sensitive write APIs, but they are not the right baseline for bulk + ingest.

+

Use the Write-Heavy Storage Preset

+

For durable file-backed ingest, start with + UseWriteOptimizedPreset().

+

That is the current recommended storage baseline for write-heavy engine + workloads and is documented in both:

+
    +
  • CSharpDB.Engine README
    • +
  • CSharpDB.Storage README
    • +
+

Prefer Monotonic Primary Keys

+

Monotonic primary keys are materially cheaper than random primary keys.

+

Why:

+
    +
  • they keep the table B-tree growing on the right edge
    • +
  • they reduce split-heavy locality loss
    • +
  • they make the primary insert path more predictable
    • +
+

The durable batching matrix in this repo continues to show that random-key + primary inserts are still a major throughput cliff relative to monotonic keys.

+

Keep Secondary Indexes Off the Hot Ingest Path When You Can

+

If you only remember one design rule besides batching, it should be this one:

+

Every extra secondary index costs you per-row write work.

+

The recent Plan 5 work materially improved duplicate-bucket maintenance and + composite-index locality, but the remaining slope on realistic insert shapes is + still mostly secondary-index maintenance rather than the PK-only bulk path.

+

If your workflow allows it, the easy win is still:

+
    +
  1. load data first
    2. +
  2. build or refresh indexes after the ingest step
    3. +
+

Use Prepared Commands at the Provider Layer

+

If you are not using the engine directly and instead go through CSharpDB.Data, + the equivalent rule is to reuse prepared commands.

+

That looks like ordinary ADO.NET:

+
    +
  • create one DbCommand
    • +
  • add parameters once
    • +
  • call Prepare()
    • +
  • update parameter values per execution
    • +
+

This is the right analogue to SQLite's prepare-once, bind-many workflow.

+

Use Concurrent Insert Mode Only for the Right Shape

+

CSharpDB does have a concurrent insert mode:

+

ImplicitInsertExecutionMode.ConcurrentWriteTransactions

+

But it is not a magic global acceleration switch.

+

The measured guidance is:

+
    +
  • Keep the default serialized insert mode for hot right-edge insert loops.
    • +
  • Measure concurrent insert mode when several tasks insert into disjoint + explicit key ranges on one shared Database.
    • +
  • If each writer needs multiple statements to commit atomically, use the + explicit write-transaction APIs instead.
    • +
+

That is the difference between using a feature because it exists and using it + because the workload shape actually matches it.

+

Treat Durable Group Commit as a Measure-First Lever

+

UseDurableGroupCommit(...) can help when several writers are already + overlapping and you are willing to trade some latency for better flush sharing.

+

But it should come after:

+
    +
  • batching
    • +
  • prepared work reuse
    • +
  • key-shape cleanup
    • +
  • avoiding unnecessary indexes
    • +
+

In other words, do not use group commit as a substitute for basic workload + hygiene.

+

The Honest Public Conclusion

+

If you are turning this into a blog post, the cleanest conclusion is:

+
    +
  • CSharpDB is now ahead of the matched SQLite baseline on the primary durable + monotonic bulk row on this runner.
    • +
  • That result is real because the comparison was aligned at the schema, + batching, and durability levels.
    • +
  • The remaining hard performance work is no longer the PK-only monotonic bulk + path.
    • +
  • The bigger remaining costs are secondary-index maintenance, random-key + locality, and hot right-edge multi-writer contention.
    • +
+

That is a much stronger statement than "CSharpDB beats SQLite" because it is + specific enough to survive scrutiny.

+

Reproducing the Main Comparisons

+

Matched durable bulk rows:

+ 
dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --durable-sql-batching-scenario BatchSweep_InsertBatch_B1000_Baseline_PkOnly_Monotonic --repeat 3 --repro
+                dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --durable-sql-batching-scenario BatchSweep_InsertBatch_B10000_Baseline_PkOnly_Monotonic --repeat 3 --repro
+                dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --sqlite-compare --repeat 3 --repro
+
+

Concurrent engine versus SQLite C API:

+ 
dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --concurrent-sqlite-capi-compare --repeat 3 --repro
+
+

Concurrent ADO.NET provider comparison:

+ 
dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --concurrent-adonet-compare --repeat 3 --repro
+
+

EF Core comparison:

+ 
dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --efcore-compare --repeat 3 --repro
+                dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --efcore-compare-hybrid-shared-connection --repeat 3 --repro
+                dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --efcore-compare-auto-open-close --repeat 3 --repro
+
+ +

If you want the lower-level internal notes behind this write-up, start here:

+
    +
  • Existing comparison guide
    • +
  • Plan 5: Raw Rows/Sec Vs SQLite
    • +
  • Benchmark suite README
    • +
  • CSharpDB.Engine README
    • +
  • CSharpDB.Storage README
    • +
+
+
+
+ + + + + diff --git a/www/css/style.css b/www/css/style.css index aaf63744..b7e46ed1 100644 --- a/www/css/style.css +++ b/www/css/style.css @@ -718,6 +718,77 @@ code { line-height: 1.7; } +.doc-content ul, +.doc-content ol { + margin: 0 0 20px 22px; + color: var(--text-secondary); +} + +.doc-content li { + margin-bottom: 8px; +} + +.doc-content blockquote { + margin: 20px 0 24px; + padding: 14px 18px; + border-left: 3px solid var(--accent); + background: var(--accent-soft); + border-radius: 0 var(--radius-md) var(--radius-md) 0; + color: var(--text-secondary); +} + +.doc-content hr { + border: 0; + border-top: 1px solid var(--border); + margin: 32px 0; +} + +.doc-content pre { + margin: 0 0 24px; + padding: 20px 24px; + overflow-x: auto; + background: var(--bg-code); + border: 1px solid var(--border); + border-radius: var(--radius-md); + box-shadow: var(--shadow-sm); +} + +.doc-content pre code { + background: none; + padding: 0; + border-radius: 0; + color: var(--text-primary); + font-size: inherit; +} + +.doc-content table { + width: 100%; + border-collapse: collapse; + margin-bottom: 24px; + font-size: 0.9rem; +} + +.doc-content table th { + text-align: left; + padding: 12px 16px; + background: var(--bg-tertiary); + border-bottom: 2px solid var(--border); + font-weight: 600; + color: var(--text-primary); + white-space: nowrap; +} + +.doc-content table td { + padding: 10px 16px; + border-bottom: 1px solid var(--border); + color: var(--text-secondary); + vertical-align: top; +} + +.doc-content table tr:hover td { + background: var(--accent-soft); +} + /* ─── Doc Table ─── */ .doc-table { width: 100%; diff --git a/www/docs/getting-started-reference.html b/www/docs/getting-started-reference.html new file mode 100644 index 00000000..5771d8e5 --- /dev/null +++ b/www/docs/getting-started-reference.html @@ -0,0 +1,688 @@ + + + + + + + + Getting Started Source Reference — CSharpDB + + + + + + + + + + + + + + +
+
+ + +
+
Source reference. This page preserves the original long-form markdown content that previously lived at docs/getting-started.md. For the shorter curated page, see Getting Started.
+

Getting Started with CSharpDB

+

This tutorial walks you through using CSharpDB from opening a database to running queries and transactions.

+

Prerequisites

+ +

Setup

+

Add a project reference to CSharpDB.Engine (which transitively pulls in all other projects):

+ 
dotnet add reference path/to/src/CSharpDB.Engine/CSharpDB.Engine.csproj
+
+

Then add the using directive:

+ 
using CSharpDB.Engine;
+
+
+

1. Opening a Database

+

Database.OpenAsync opens an existing database file or creates a new one if it doesn't exist:

+ 
await using var db = await Database.OpenAsync("myapp.db");
+
+

The await using pattern ensures the database is properly closed when you're done. On open, if a WAL (Write-Ahead Log) file exists from a previous crash, the database automatically recovers committed data.

+
+

2. Creating Tables

+ 
await db.ExecuteAsync(@"
+                    CREATE TABLE products (
+                        id INTEGER PRIMARY KEY,
+                        name TEXT NOT NULL,
+                        price REAL,
+                        category TEXT
+                    )
+                ");
+
+

Supported column types:

+
    +
  • INTEGER — 64-bit signed integer (long)
    • +
  • REAL — 64-bit floating point (double)
    • +
  • TEXT — UTF-8 string
    • +
  • BLOB — raw byte array
    • +
+

Column constraints:

+
    +
  • PRIMARY KEY — marks the column as the primary key
    • +
  • IDENTITY / AUTOINCREMENT — mark an INTEGER PRIMARY KEY as an identity column (explicit ID inserts are still allowed)
    • +
  • NOT NULL — disallows NULL values
    • +
+

Use IF NOT EXISTS to avoid errors when the table already exists:

+ 
await db.ExecuteAsync("CREATE TABLE IF NOT EXISTS products (id INTEGER, name TEXT)");
+
+
+

3. Inserting Data

+

Insert with all columns:

+ 
await db.ExecuteAsync("INSERT INTO products VALUES (1, 'Widget', 9.99, 'Hardware')");
+                await db.ExecuteAsync("INSERT INTO products VALUES (2, 'Gadget', 29.99, 'Electronics')");
+                await db.ExecuteAsync("INSERT INTO products VALUES (3, 'Doohickey', 4.99, 'Hardware')");
+
+

Insert with named columns (unspecified columns get NULL):

+ 
await db.ExecuteAsync("INSERT INTO products (id, name) VALUES (4, 'Thingamajig')");
+
+

The return value of ExecuteAsync for DML statements includes a RowsAffected count:

+ 
var result = await db.ExecuteAsync("INSERT INTO products VALUES (5, 'Gizmo', 14.99, 'Electronics')");
+                Console.WriteLine($"Inserted {result.RowsAffected} row(s)"); // Inserted 1 row(s)
+
+
+

4. Querying Data

+

SELECT all rows

+ 
await using var result = await db.ExecuteAsync("SELECT * FROM products");
+                
+                await foreach (var row in result.GetRowsAsync())
+                {
+                    long id = row[0].AsInteger;
+                    string name = row[1].AsText;
+                    Console.WriteLine($"  {id}: {name}");
+                }
+
+
+

Important: Use await using on SELECT results to properly dispose the underlying operator chain.

+
+

Materialize all rows at once

+

If you want all rows in a list instead of streaming:

+ 
await using var result = await db.ExecuteAsync("SELECT * FROM products");
+                var rows = await result.ToListAsync();
+                
+                Console.WriteLine($"Got {rows.Count} products");
+
+

WHERE clause

+ 
await using var result = await db.ExecuteAsync(
+                    "SELECT * FROM products WHERE price > 10.0 AND category = 'Electronics'");
+
+

Supported operators in WHERE: =, <>, <, >, <=, >=, AND, OR, NOT, +, -, *, /, LIKE, IN, BETWEEN, IS NULL, IS NOT NULL

+

LIKE pattern matching

+ 
// % matches any sequence, _ matches any single character
+                await using var result = await db.ExecuteAsync(
+                    "SELECT * FROM products WHERE name LIKE 'Gad%'");
+
+

IN lists

+ 
await using var result = await db.ExecuteAsync(
+                    "SELECT * FROM products WHERE category IN ('Hardware', 'Electronics')");
+
+

BETWEEN ranges

+ 
await using var result = await db.ExecuteAsync(
+                    "SELECT * FROM products WHERE price BETWEEN 5.0 AND 20.0");
+
+

ORDER BY

+ 
await using var result = await db.ExecuteAsync(
+                    "SELECT * FROM products ORDER BY price DESC");
+
+

You can sort by multiple columns and mix ASC/DESC:

+ 
await using var result = await db.ExecuteAsync(
+                    "SELECT * FROM products ORDER BY category ASC, price DESC");
+
+

LIMIT and OFFSET

+ 
await using var result = await db.ExecuteAsync(
+                    "SELECT * FROM products ORDER BY price DESC LIMIT 3");
+                
+                // Skip the first 5, then take the next 10
+                await using var paged = await db.ExecuteAsync(
+                    "SELECT * FROM products ORDER BY id LIMIT 10 OFFSET 5");
+
+

Select specific columns

+ 
await using var result = await db.ExecuteAsync(
+                    "SELECT name, price FROM products WHERE price > 5.0");
+
+

System catalog metadata (sys.*)

+

You can inspect tables, columns, indexes, foreign keys, views, triggers, and object inventory with SQL:

+ 
await using var tables = await db.ExecuteAsync(
+                    "SELECT table_name, column_count, primary_key_column FROM sys.tables ORDER BY table_name");
+                
+                await using var columns = await db.ExecuteAsync(
+                    "SELECT column_name, data_type, is_nullable FROM sys.columns " +
+                    "WHERE table_name = 'products' ORDER BY ordinal_position");
+                
+                await using var objects = await db.ExecuteAsync(
+                    "SELECT object_name, object_type, parent_table_name FROM sys.objects ORDER BY object_type, object_name");
+
+

Catalog sources:

+
    +
  • sys.tables
    • +
  • sys.columns
    • +
  • sys.indexes
    • +
  • sys.foreign_keys
    • +
  • sys.views
    • +
  • sys.triggers
    • +
  • sys.objects
    • +
  • sys.saved_queries
    • +
+

Underscored aliases are also available (sys_tables, sys_columns, etc.).

+
+

5. Aggregate Functions

+ 
// COUNT
+                await using var r1 = await db.ExecuteAsync("SELECT COUNT(*) FROM products");
+                
+                // SUM, AVG, MIN, MAX
+                await using var r2 = await db.ExecuteAsync(
+                    "SELECT category, COUNT(*), AVG(price), MIN(price), MAX(price) FROM products GROUP BY category");
+                
+                // HAVING
+                await using var r3 = await db.ExecuteAsync(
+                    "SELECT category, COUNT(*) as cnt FROM products GROUP BY category HAVING cnt > 1");
+
+

Supported aggregate functions: COUNT(*), COUNT(col), COUNT(DISTINCT col), SUM, AVG, MIN, MAX

+
+

6. JOINs

+ 
await db.ExecuteAsync("CREATE TABLE orders (id INTEGER, product_id INTEGER, qty INTEGER)");
+                await db.ExecuteAsync("INSERT INTO orders VALUES (1, 1, 10)");
+                await db.ExecuteAsync("INSERT INTO orders VALUES (2, 2, 5)");
+                
+                // INNER JOIN
+                await using var result = await db.ExecuteAsync(@"
+                    SELECT p.name, o.qty
+                    FROM products p
+                    INNER JOIN orders o ON p.id = o.product_id");
+                
+                // LEFT JOIN (all products, even those without orders)
+                await using var result2 = await db.ExecuteAsync(@"
+                    SELECT p.name, o.qty
+                    FROM products p
+                    LEFT JOIN orders o ON p.id = o.product_id");
+
+

Supported join types: INNER JOIN, LEFT JOIN, RIGHT JOIN, CROSS JOIN

+
+

7. Updating Rows

+ 
var result = await db.ExecuteAsync(
+                    "UPDATE products SET price = 12.99 WHERE name = 'Widget'");
+                
+                Console.WriteLine($"Updated {result.RowsAffected} row(s)");
+
+

You can update multiple columns at once:

+ 
await db.ExecuteAsync(
+                    "UPDATE products SET price = 19.99, category = 'Premium' WHERE id = 2");
+
+
+

8. Deleting Rows

+ 
var result = await db.ExecuteAsync("DELETE FROM products WHERE category = 'Hardware'");
+                Console.WriteLine($"Deleted {result.RowsAffected} row(s)");
+
+

Delete all rows (no WHERE clause):

+ 
await db.ExecuteAsync("DELETE FROM products");
+
+
+

9. Working with NULL

+

Insert a NULL value explicitly:

+ 
await db.ExecuteAsync("INSERT INTO products VALUES (10, 'Mystery', NULL, NULL)");
+
+

Check for NULL when reading:

+ 
await using var result = await db.ExecuteAsync("SELECT * FROM products");
+                await foreach (var row in result.GetRowsAsync())
+                {
+                    if (row[2].IsNull)
+                        Console.WriteLine($"{row[1].AsText}: no price set");
+                    else
+                        Console.WriteLine($"{row[1].AsText}: ${row[2].AsReal}");
+                }
+
+

Filter with IS NULL / IS NOT NULL:

+ 
await using var result = await db.ExecuteAsync(
+                    "SELECT * FROM products WHERE price IS NOT NULL");
+
+
+

10. Transactions

+

By default, each DML/DDL statement auto-commits. For multi-statement atomicity, use explicit transactions:

+ 
await db.BeginTransactionAsync();
+                try
+                {
+                    await db.ExecuteAsync("INSERT INTO products VALUES (20, 'Item A', 5.00, 'Batch')");
+                    await db.ExecuteAsync("INSERT INTO products VALUES (21, 'Item B', 7.50, 'Batch')");
+                    await db.ExecuteAsync("INSERT INTO products VALUES (22, 'Item C', 3.25, 'Batch')");
+                
+                    await db.CommitAsync(); // All three inserts are now durable
+                }
+                catch
+                {
+                    await db.RollbackAsync(); // None of the inserts are persisted
+                    throw;
+                }
+
+

Transactions also improve performance for bulk inserts since dirty pages are written to the WAL only once rather than per-statement.

+

Rollback example

+ 
await db.BeginTransactionAsync();
+                await db.ExecuteAsync("DELETE FROM products"); // All rows deleted in memory
+                await db.RollbackAsync();                      // Changes discarded — all rows restored
+
+
+

11. Indexes

+

Create indexes to speed up equality lookups:

+ 
// Regular index
+                await db.ExecuteAsync("CREATE INDEX idx_category ON products (category)");
+                
+                // Unique index (enforces uniqueness)
+                await db.ExecuteAsync("CREATE UNIQUE INDEX idx_name ON products (name)");
+
+

When a WHERE clause contains column = value on an indexed column, the query planner automatically uses the index instead of a full table scan.

+
+

12. ALTER TABLE

+ 
// Add a column
+                await db.ExecuteAsync("ALTER TABLE products ADD COLUMN weight REAL");
+                
+                // Drop a column
+                await db.ExecuteAsync("ALTER TABLE products DROP COLUMN weight");
+                
+                // Drop a foreign key constraint by name
+                await db.ExecuteAsync("ALTER TABLE child_orders DROP CONSTRAINT fk_child_orders_parent_id_abcd1234");
+                
+                // Rename a column
+                await db.ExecuteAsync("ALTER TABLE products RENAME COLUMN category TO department");
+                
+                // Rename a table
+                await db.ExecuteAsync("ALTER TABLE products RENAME TO inventory");
+
+

Use sys.foreign_keys to discover generated foreign key names before dropping one:

+ 
await using var foreignKeys = await db.ExecuteAsync(
+                    "SELECT constraint_name, table_name, column_name FROM sys.foreign_keys ORDER BY table_name, column_name");
+
+

Retrofitting foreign keys onto older databases

+

If you already have tables on disk without REFERENCES metadata, opening the database on a newer engine does not add foreign keys automatically. Use the maintenance migration workflow when you want to validate and then persist FK metadata onto existing tables:

+ 
using CSharpDB.Client;
+                using CSharpDB.Client.Models;
+                
+                await using var client = CSharpDbClient.Create(new CSharpDbClientOptions
+                {
+                    DataSource = "myapp.db"
+                });
+                
+                var spec =
+                    new[]
+                    {
+                        new ForeignKeyMigrationConstraintSpec
+                        {
+                            TableName = "orders",
+                            ColumnName = "customer_id",
+                            ReferencedTableName = "customers",
+                            ReferencedColumnName = "id",
+                            OnDelete = ForeignKeyOnDeleteAction.Cascade,
+                        },
+                    };
+                
+                var preview = await client.MigrateForeignKeysAsync(new ForeignKeyMigrationRequest
+                {
+                    ValidateOnly = true,
+                    ViolationSampleLimit = 100,
+                    Constraints = spec,
+                });
+                
+                if (!preview.Succeeded)
+                {
+                    foreach (var violation in preview.Violations)
+                        Console.WriteLine($"{violation.TableName}.{violation.ColumnName}: {violation.Reason}");
+                }
+                else
+                {
+                    await client.MigrateForeignKeysAsync(new ForeignKeyMigrationRequest
+                    {
+                        BackupDestinationPath = "pre-fk.backup.db",
+                        Constraints = spec,
+                    });
+                }
+
+

Notes:

+
    +
  • ValidateOnly = true previews the migration without changing schema or data.
    • +
  • BackupDestinationPath is optional but recommended for apply mode.
    • +
  • The same operation is available through HTTP, gRPC, the CLI, and the Admin Storage tab.
    • +
+

Schema migration pattern (PRIMARY KEY / IDENTITY changes)

+

ALTER TABLE does not currently support changing an existing column to/from PRIMARY KEY or IDENTITY. + Use a create-copy-swap migration:

+ 
BEGIN TRANSACTION;
+                
+                -- 1) Create target schema
+                CREATE TABLE users_v2 (
+                    id INTEGER PRIMARY KEY IDENTITY,
+                    name TEXT NOT NULL,
+                    email TEXT NOT NULL
+                );
+                
+                -- 2) Copy data
+                -- For small tables, insert explicit values:
+                INSERT INTO users_v2 (id, name, email) VALUES (1, 'Alice', 'alice@acme.io');
+                INSERT INTO users_v2 (id, name, email) VALUES (2, 'Bob', 'bob@acme.io');
+                
+                -- 3) Swap tables
+                DROP TABLE users;
+                ALTER TABLE users_v2 RENAME TO users;
+                
+                COMMIT;
+
+

Notes:

+
    +
  • Explicit identity values are allowed, so preserving old IDs is supported.
    • +
  • If you omit id during insert, CSharpDB auto-generates identity values.
    • +
  • Bulk INSERT INTO ... SELECT ... copy is not yet supported, so large-table copy should be done via application code (read rows, then insert into the new table).
    • +
+
+

13. Views

+

Views are named, reusable queries:

+ 
await db.ExecuteAsync(@"
+                    CREATE VIEW expensive_products AS
+                    SELECT name, price FROM products WHERE price > 20.0");
+                
+                // Query the view like a table
+                await using var result = await db.ExecuteAsync("SELECT * FROM expensive_products");
+
+
+

14. Common Table Expressions (CTEs)

+ 
await using var result = await db.ExecuteAsync(@"
+                    WITH high_value AS (
+                        SELECT * FROM products WHERE price > 10.0
+                    )
+                    SELECT name, price FROM high_value ORDER BY price DESC");
+
+
+

15. Triggers

+

Triggers execute SQL automatically when data changes:

+ 
await db.ExecuteAsync("CREATE TABLE audit_log (action TEXT, product_name TEXT)");
+                
+                await db.ExecuteAsync(@"
+                    CREATE TRIGGER log_insert
+                    AFTER INSERT ON products
+                    BEGIN
+                        INSERT INTO audit_log VALUES ('INSERT', NEW.name);
+                    END");
+                
+                // Now inserting into products automatically logs to audit_log
+                await db.ExecuteAsync("INSERT INTO products VALUES (50, 'Auto-logged', 9.99, 'Test')");
+
+

Supported trigger types:

+
    +
  • BEFORE INSERT, AFTER INSERT
    • +
  • BEFORE UPDATE, AFTER UPDATE
    • +
  • BEFORE DELETE, AFTER DELETE
    • +
+

Use NEW.column in INSERT/UPDATE triggers and OLD.column in UPDATE/DELETE triggers.

+
+

16. Concurrent Readers

+

Create reader sessions for snapshot-isolated reads that don't block writes:

+ 
// Take a snapshot of the current database state
+                using var reader = db.CreateReaderSession();
+                
+                // Writer can continue modifying data — reader won't see changes
+                await db.ExecuteAsync("INSERT INTO products VALUES (99, 'New item', 1.0, 'Test')");
+                
+                // Reader sees the database as it was when the snapshot was taken
+                await using var result = await reader.ExecuteReadAsync("SELECT COUNT(*) FROM products");
+
+

Multiple reader sessions can be active simultaneously.

+
+

17. Dropping Tables

+ 
await db.ExecuteAsync("DROP TABLE products");
+
+

Use IF EXISTS to avoid errors:

+ 
await db.ExecuteAsync("DROP TABLE IF EXISTS products");
+
+
+

18. Error Handling

+

CSharpDB throws CSharpDbException with a typed ErrorCode:

+ 
using CSharpDB.Primitives;
+                
+                try
+                {
+                    await db.ExecuteAsync("SELECT * FROM nonexistent");
+                }
+                catch (CSharpDbException ex) when (ex.Code == ErrorCode.TableNotFound)
+                {
+                    Console.WriteLine($"Table not found: {ex.Message}");
+                }
+                catch (CSharpDbException ex) when (ex.Code == ErrorCode.SyntaxError)
+                {
+                    Console.WriteLine($"SQL syntax error: {ex.Message}");
+                }
+
+

Error codes:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CodeMeaning
TableNotFoundReferenced table doesn't exist
TableAlreadyExistsCREATE TABLE on an existing table (without IF NOT EXISTS)
ColumnNotFoundReferenced column doesn't exist in the table
DuplicateKeyINSERT with a rowid that already exists
SyntaxErrorInvalid SQL syntax
TypeMismatchValue type doesn't match expected type
ConstraintViolationNOT NULL, UNIQUE, or other constraint violated
IoErrorFile system read/write failure
CorruptDatabaseDatabase file structure is invalid
WalErrorError reading/writing the WAL file
BusyCould not acquire write lock (another writer is active)
+
+

19. Using the ADO.NET Provider

+

For standard .NET data access patterns, use the CSharpDB.Data package:

+ 
using CSharpDB.Data;
+                
+                await using var conn = new CSharpDbConnection("Data Source=myapp.db");
+                await conn.OpenAsync();
+                
+                // Parameterized queries
+                using var cmd = conn.CreateCommand();
+                cmd.CommandText = "SELECT * FROM products WHERE price > @minPrice";
+                cmd.Parameters.AddWithValue("@minPrice", 10.0);
+                
+                await using var reader = await cmd.ExecuteReaderAsync();
+                while (await reader.ReadAsync())
+                {
+                    long id = reader.GetInt64(0);
+                    string name = reader.GetString(1);
+                    double price = reader.GetDouble(2);
+                    Console.WriteLine($"{id}: {name} (${price})");
+                }
+
+

The current provider routes ordinary embedded and daemon-backed usage through + the same authoritative CSharpDB.Client layer used by the other host surfaces. + That means you keep one provider API shape while choosing either a local + embedded database or a remote daemon endpoint in the connection string.

+

ExecuteScalar

+ 
cmd.CommandText = "SELECT COUNT(*) FROM products";
+                var count = await cmd.ExecuteScalarAsync();
+
+

ExecuteNonQuery

+ 
cmd.CommandText = "INSERT INTO products VALUES (100, 'New', 5.99, 'Test')";
+                int rowsAffected = await cmd.ExecuteNonQueryAsync();
+
+

Transactions via ADO.NET

+ 
await using var txn = await conn.BeginTransactionAsync();
+                try
+                {
+                    using var cmd = conn.CreateCommand();
+                    cmd.Transaction = (CSharpDbTransaction)txn;
+                    cmd.CommandText = "INSERT INTO products VALUES (200, 'Txn Item', 1.0, 'Test')";
+                    await cmd.ExecuteNonQueryAsync();
+                    await txn.CommitAsync();
+                }
+                catch
+                {
+                    await txn.RollbackAsync();
+                    throw;
+                }
+
+

Daemon-Backed ADO.NET

+

If you want the same ADO.NET surface against a long-lived CSharpDB.Daemon + process, change only the connection string:

+ 
using CSharpDB.Data;
+                
+                await using var conn = new CSharpDbConnection(
+                    "Transport=Grpc;Endpoint=http://localhost:5820");
+                await conn.OpenAsync();
+                
+                using var cmd = conn.CreateCommand();
+                cmd.CommandText = "SELECT COUNT(*) FROM products";
+                var count = (long)(await cmd.ExecuteScalarAsync() ?? 0L);
+
+

Notes:

+
    +
  • embedded/local mode uses Data Source=...
    • +
  • daemon-backed mode uses Transport=Grpc;Endpoint=http://...
    • +
  • named shared in-memory connections such as Data Source=:memory:shared are process-local and do not cross the daemon boundary
    • +
  • NamedPipes is not implemented end to end yet, so use Grpc for daemon access
    • +
+
+

20. Data Persistence

+

Data survives application restarts — it's written to disk via the WAL on commit:

+ 
// Session 1: Create and populate
+                await using (var db = await Database.OpenAsync("persistent.db"))
+                {
+                    await db.ExecuteAsync("CREATE TABLE notes (id INTEGER, text TEXT)");
+                    await db.ExecuteAsync("INSERT INTO notes VALUES (1, 'Remember this')");
+                }
+                
+                // Session 2: Data is still there
+                await using (var db = await Database.OpenAsync("persistent.db"))
+                {
+                    await using var result = await db.ExecuteAsync("SELECT * FROM notes");
+                    var rows = await result.ToListAsync();
+                    Console.WriteLine(rows[0][1].AsText); // "Remember this"
+                }
+
+
+

Next Steps

+ +
+
+
+ + + + + diff --git a/www/docs/getting-started.html b/www/docs/getting-started.html index 97d104d1..0d7c1f15 100644 --- a/www/docs/getting-started.html +++ b/www/docs/getting-started.html @@ -45,6 +45,9 @@

Getting Started

Get CSharpDB up and running in your .NET project in minutes. Choose the API that fits your needs.

+
+ Need the full walkthrough? The original step-by-step markdown guide is preserved as Getting Started Source Reference. +

Installation

diff --git a/www/docs/internals.html b/www/docs/internals.html index 38774628..7e781f08 100644 --- a/www/docs/internals.html +++ b/www/docs/internals.html @@ -204,8 +204,7 @@

Project Structure

├── docs/ │ ├── tutorials/native-ffi/ FFI tutorials (JavaScript via koffi, Python via ctypes) │ ├── tutorials/storage/ Storage tutorial track, study examples, and advanced standalone examples -│ ├── roadmap.md Product roadmap and status -│ └── rest-api.md REST host reference +│ └── roadmap.md Product roadmap and status │ └── samples/ Sample datasets + import helpers ├── ecommerce-store/ diff --git a/www/docs/performance-reference.html b/www/docs/performance-reference.html new file mode 100644 index 00000000..be35a17c --- /dev/null +++ b/www/docs/performance-reference.html @@ -0,0 +1,662 @@ + + + + + + + + Performance Guide Source Reference — CSharpDB + + + + + + + + + + + + + + +
+
+ + +
+
Source reference. This page preserves the original long-form markdown content that previously lived at docs/performance.md. For the shorter curated page, see Performance Guide.
+

CSharpDB Performance Guide

+

This guide is about getting real performance out of CSharpDB, not just turning knobs. + It is organized by workload scenario and grounded in the current benchmark and documentation set in this repo.

+

Benchmark Basis

+

This guide uses the latest current sources:

+
    +
  • The latest perf guardrail report on March 29, 2026 passed 184/184 checks on the benchmark runner.
    • +
  • The broader macro, direct-client, and hybrid comparison captures were refreshed on March 28, 2026.
    • +
  • The published micro spot checks in tests/CSharpDB.Benchmarks/README.md and BenchmarkDotNet.Artifacts/results/ provide the query-shape details used below.
    • +
  • The benchmark README still includes some earlier March 29 narrative from a failing midday rerun; this guide uses the latest passing guardrail report as the current benchmark status.
    • +
+

Benchmark environment for the published numbers:

+
    +
  • CPU: Intel Core i9-11900K
    • +
  • OS: Windows 11 (10.0.26300 benchmark runner)
    • +
  • Runtime: .NET 10
    • +
  • Disk: NVMe SSD
    • +
+

Treat the numbers here as directional for decision-making, then rerun the benchmark suite on your hardware before locking in a tuning choice.

+

Start Here

+

Use this table first. Most CSharpDB performance wins come from choosing the right mode and access path before touching advanced settings.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ScenarioStart withWhy
Durable local app, mostly point readsFile-backed + UseDirectLookupOptimizedPreset()Best simple baseline for hot local reads
Durable app with a known hot working setHybrid incremental-durable + HotTableNames / HotCollectionNamesBest when you can pay open cost once and then hammer the same hot objects
Ephemeral cache or periodic snapshot workflowIn-memory + SaveToFileAsync when neededBiggest write throughput win by far
Burst of related SQL readsReuse one ReaderSession per reader burstSnapshot reuse is much cheaper than per-query session creation
Ordered/range SQL queriesBuild indexes for the filter/sort columns and project only what you needCovered and compact paths are major wins
Join/reporting workloadAdd join indexes and run ANALYZE after bulk changesPlanner can use histograms, heavy hitters, composite-prefix stats, build-side choice, selective lookups, and bounded small-chain reordering
Write-heavy ingestUseWriteOptimizedPreset() plus explicit transaction batchingBatching is the biggest durable-write lever in the current suite
Document/path queriesCollection<T> + EnsureIndexAsync(...) on fields and paths you queryIndexed collection paths are fast; unindexed predicate scans are not
ADO.NET app with frequent opensEnable pooling explicitlyPooling dominates open/close overhead in the current provider benchmarks
+

The Biggest Levers

+

If you remember only a few rules, use these:

+
    +
  1. Choose the right storage mode first.
    2. +
  2. Batch durable writes instead of auto-committing row-by-row.
    3. +
  3. Reuse ReaderSession for bursts of related SQL reads.
    4. +
  4. Create indexes that match the filter, join, and ORDER BY shape.
    5. +
  5. Prefer covered or narrow projections over SELECT *.
    6. +
  6. Run ANALYZE after bulk loads or major data-distribution changes.
    7. +
  7. Measure before enabling advanced knobs like caching indexes, durable batch windows, or WAL preallocation.
    8. +
+

Scenario 1: Hot Point Reads In A Local Embedded App

+

This is the default "application database" scenario: the app is already open, the database is local, and the hot path is GetAsync, primary-key lookups, or small indexed reads.

+

What to do

+
    +
  • Start file-backed with UseDirectLookupOptimizedPreset().
    • +
  • Use Collection<T>.GetAsync(...) for pure key/document access.
    • +
  • Use SELECT ... WHERE id = ... or equivalent indexed equality lookups for relational paths.
    • +
  • Reuse a ReaderSession when you are doing many related reads from the same snapshot.
    • +
+

Why

+

Current published steady-state numbers are already strong without exotic tuning:

+
    +
  • File-backed SQL point lookups in the hot steady-state harness were about 1.10M ops/sec.
    • +
  • File-backed collection point gets were about 1.78M ops/sec.
    • +
  • Hot-cache micro references in the benchmark README still reach roughly 4.16M SQL PK lookups/sec and 2.63M collection gets/sec on the benchmark runner.
    • +
+

For direct file-backed opens, the benchmark README's current recommendation is:

+
    +
  • builder.UseDirectLookupOptimizedPreset() for hot local lookup workloads
    • +
+ 
using CSharpDB.Engine;
+                
+                var options = new DatabaseOptions()
+                    .ConfigureStorageEngine(builder => builder.UseDirectLookupOptimizedPreset());
+                
+                await using var db = await Database.OpenAsync("app.db", options);
+                
+                await using var result = await db.ExecuteAsync(
+                    "SELECT id, name FROM users WHERE id = 42");
+
+

When to use Collection<T>

+

Use the collection API when the workload is fundamentally:

+
    +
  • key by id
    • +
  • document get/put/delete
    • +
  • indexed field equality
    • +
  • indexed path equality/range
    • +
+

That path skips the SQL front door entirely.

+

Scenario 2: Burst SQL Reads While Writes Continue

+

This is the common local service pattern: a writer is active, but readers want a stable snapshot and tend to issue several related queries together.

+

What to do

+
    +
  • Create one ReaderSession per concurrent reader.
    • +
  • Reuse that session for a burst of related reads.
    • +
  • Dispose each QueryResult before issuing the next query on the same session.
    • +
+ 
using var reader = db.CreateReaderSession();
+                
+                await using (var users = await reader.ExecuteReadAsync(
+                    "SELECT id, name FROM users WHERE tier = 'gold'"))
+                {
+                    while (await users.MoveNextAsync())
+                    {
+                    }
+                }
+                
+                await using (var orders = await reader.ExecuteReadAsync(
+                    "SELECT id, total FROM orders WHERE customer_id = 42"))
+                {
+                    while (await orders.MoveNextAsync())
+                    {
+                    }
+                }
+
+

Why

+

The current dedicated ReaderSessionBenchmarks show the setup penalty clearly:

+
    +
  • per-query reader-session point lookups are roughly 3x slower than reused-session lookups
    • +
  • reused reader-session lookups are effectively at the same floor as direct ExecuteAsync
    • +
+

The practical reading is simple:

+
    +
  • ReaderSession is the right tool for concurrent snapshot reads.
    • +
  • Reusing it is almost always better than recreating it for every small query.
    • +
+

Scenario 3: Ordered And Range SQL Queries

+

This is where query shape matters a lot. The main question is whether the engine can stay on index data, or whether it has to keep fetching wide base rows.

+

What to do

+
    +
  • Index the filter or sort column.
    • +
  • If possible, project only the indexed columns plus the row id.
    • +
  • Prefer narrow projections over SELECT *.
    • +
  • For multi-column filters, use composite indexes that match the left-to-right predicate shape.
    • +
+ 
CREATE INDEX idx_orders_created_at ON orders(created_at);
+                
+                SELECT id, created_at
+                FROM orders
+                WHERE created_at BETWEEN @from AND @to
+                ORDER BY created_at
+                LIMIT 100;
+
+

Why

+

The current OrderByIndexBenchmarks and README spot checks show large wins when the query stays on index data:

+
    +
  • ORDER BY value LIMIT 100 on 100K rows: +
      +
    • index-order scan: about 64.79 us
      • +
    • covered index-order scan: about 23.17 us
      • +
    +
    • +
  • WHERE value BETWEEN ... on 100K rows: +
      +
    • row fetch path: about 59.48 ms
      • +
    • covered projection: about 18.49 ms
      • +
    +
    • +
  • Composite covered projection on 100K rows: +
      +
    • about 2.523 us
      • +
    +
    • +
+

This is one of the clearest themes in the repo's benchmarks:

+
    +
  • the engine is fast at finding qualifying rows
    • +
  • the expensive part is often row fetch and materialization
    • +
  • covered and compact projections are a major lever
    • +
+

Practical rule

+

If the business endpoint only needs id, status, and created_at, do not ask CSharpDB to materialize twelve more columns.

+

Scenario 4: Reporting, Joins, And Selective Filters

+

This is the "real SQL" scenario: non-trivial joins, selective predicates, and planner choice starts to matter.

+

What to do

+
    +
  • Create indexes on join keys and on selective filter columns.
    • +
  • Run ANALYZE after bulk loads and after major distribution changes.
    • +
  • Inspect sys.table_stats and sys.column_stats when a plan is not behaving as expected.
    • +
+ 
CREATE INDEX idx_orders_customer ON orders(customer_id);
+                CREATE INDEX idx_order_items_order ON order_items(order_id);
+                
+                ANALYZE;
+                
+                SELECT * FROM sys.table_stats ORDER BY table_name;
+                SELECT * FROM sys.column_stats ORDER BY table_name, ordinal_position;
+
+

Why

+

Current planner work already uses persisted stats for:

+
    +
  • selective lookup choice
    • +
  • join method choice
    • +
  • hash build-side choice
    • +
  • limited inner-join reordering
    • +
+

The benchmarks show this matters:

+
    +
  • INNER JOIN 1Kx20K (planner swap build side): about 7.16 ms
    • +
  • INNER JOIN 1Kx20K (no swap via view): about 11.43 ms
    • +
  • INNER JOIN on right PK (index nested-loop): about 482.70 us
    • +
  • INNER JOIN on right PK (forced hash): about 833.54 us
    • +
+

The practical rule:

+
    +
  • if a reporting or join workload matters, index it and analyze it
    • +
  • if you skip ANALYZE, you are leaving planner quality on the table
    • +
+

Scenario 5: Durable Write-Heavy Ingest

+

This is where the wrong default usage hurts most. If you write one durable row at a time, the fixed WAL/fsync cost dominates.

+

What to do

+
    +
  • Start with UseWriteOptimizedPreset().
    • +
  • Batch writes in explicit transactions.
    • +
  • Measure with your real batch size before touching advanced durable-write knobs.
    • +
+ 
using CSharpDB.Engine;
+                
+                var options = new DatabaseOptions()
+                    .ConfigureStorageEngine(builder => builder.UseWriteOptimizedPreset());
+                
+                await using var db = await Database.OpenAsync("ingest.db", options);
+                
+                await db.BeginTransactionAsync();
+                try
+                {
+                    foreach (var row in rows)
+                    {
+                        await db.ExecuteAsync(
+                            $"INSERT INTO events VALUES ({row.Id}, {row.TimestampTicks})");
+                    }
+                
+                    await db.CommitAsync();
+                }
+                catch
+                {
+                    await db.RollbackAsync();
+                    throw;
+                }
+
+

Why

+

The current durable batching benchmarks show that batching is the largest lever in the repo:

+
    +
  • auto-commit single-row SQL: about 270.5 ops/sec
    • +
  • explicit transaction, 10 rows/commit: about 2,695.8 rows/sec
    • +
  • explicit transaction, 100 rows/commit: about 26,999.5 rows/sec
    • +
  • explicit transaction, 1000 rows/commit: about 197,256.9 rows/sec
    • +
+

That is the core write-performance message for CSharpDB:

+
    +
  • if you can batch, batch
    • +
  • if you cannot batch, expect durable single-row throughput to be much lower
    • +
+

Preset guidance

+

Current repo guidance is:

+
    +
  • start with UseWriteOptimizedPreset()
    • +
  • do not assume UseLowLatencyDurableWritePreset() is faster on your workload
    • +
+

On the current benchmark runner:

+
    +
  • analyzed UseWriteOptimizedPreset(): about 267.8 ops/sec
    • +
  • analyzed UseLowLatencyDurableWritePreset(): about 261.4 ops/sec
    • +
+

So the "low latency" preset is measure-first, not the default recommendation.

+

Scenario 6: Multi-Writer Durable Contention

+

This is narrower than normal ingest: several writer tasks are hitting the same shared Database instance.

+

What to do

+
    +
  • Keep the baseline at UseWriteOptimizedPreset().
    • +
  • For 8 in-process writers, benchmark WAL preallocation first.
    • +
  • For 4 in-process writers, benchmark a small durable batch window first.
    • +
  • Do not cargo-cult batch-window tuning from the shared-writer case into the single-writer case.
    • +
+

Why

+

The current concurrent write diagnostics say:

+
    +
  • best 8-writer row: W8_Batch0_Prealloc1MiB at about 1078.6 commits/sec
    • +
  • close followers: W8_Batch250us at 1070.4, W8_Batch0 at 1068.2
    • +
  • best 4-writer row: W4_Batch250us at about 553.4 commits/sec
    • +
+

But the single-writer durable diagnostics say the opposite for batch windows:

+
    +
  • BatchWindow(250us): about 267.2 ops/sec
    • +
  • BatchWindow(1ms): about 65.8 ops/sec
    • +
+

So:

+
    +
  • shared-writer contention can justify small wait windows or preallocation
    • +
  • single-writer durable paths usually cannot
    • +
+

Scenario 7: Cold File Reads, Tooling, And Cache-Pressured Workloads

+

This is the "open a big local file and probe around" scenario: admin tools, inspection tools, one-shot analytics, or workloads that do not stay hot.

+

What to do

+
    +
  • Start with UseDirectColdFileLookupPreset() for direct file-backed opens.
    • +
  • If your hot pages are still living in the WAL, benchmark a small WAL read cache.
    • +
  • For deliberately bounded-cache workloads, start around UseHybridFileCachePreset(2048) scale rather than tiny caches.
    • +
+ 
var options = new DatabaseOptions()
+                    .ConfigureStorageEngine(builder => builder.UseDirectColdFileLookupPreset());
+                
+                await using var db = await Database.OpenAsync("cold.db", options);
+
+

Why

+

The current cold-read benchmarks show:

+
    +
  • SQL cold lookup, copy-based path: 28.704 us
    • +
  • SQL cold lookup, WAL-backed with 128-page WAL cache: 19.83 us
    • +
+

The current file-backed collection tuning matrix also shows cache size matters:

+
    +
  • indexed collection lookup at 16 pages: 54.15 us
    • +
  • indexed collection lookup at 2048 pages: 20.57 us
    • +
+

Two practical takeaways:

+
    +
  • cold local reads benefit from the mapped-file read path
    • +
  • tiny bounded caches are expensive unless the workload is truly memory-constrained
    • +
+

What not to do

+

Do not assume caching indexes are automatically a win.

+

The current tuning matrix found UseCachingIndexes neutral-to-negative on these lookup workloads, including a case where a reused reader-session SQL lookup worsened from 41.36 us to 212.94 us.

+

Scenario 8: Known Hot Set With Durable Backing

+

This is the hybrid-mode sweet spot: the database must stay durable on disk, but the same tables or collections are hot enough that prewarming them once is worth it.

+

What to do

+
    +
  • Use OpenHybridAsync(...) with IncrementalDurable.
    • +
  • Set HotTableNames and/or HotCollectionNames for the truly hot objects.
    • +
  • Use this only when the process is long-lived enough to amortize open cost.
    • +
+ 
await using var db = await Database.OpenHybridAsync(
+                    "app.db",
+                    new DatabaseOptions(),
+                    new HybridDatabaseOptions
+                    {
+                        PersistenceMode = HybridPersistenceMode.IncrementalDurable,
+                        HotTableNames = ["users", "orders"],
+                        HotCollectionNames = ["sessions"]
+                    });
+
+

Why

+

In the dedicated resident hot-set harness:

+
    +
  • hybrid hot-set incremental-durable SQL burst: 625.76K ops/sec
    • +
  • hybrid hot-set incremental-durable collection burst: 707.38K ops/sec
    • +
+

But the trade-off is open cost:

+
    +
  • hybrid hot-set SQL open only: about 87.39 ms
    • +
  • hybrid hot-set collection open only: about 128.59 ms
    • +
+

This mode is excellent when:

+
    +
  • open happens once
    • +
  • the process stays up
    • +
  • the same objects stay hot
    • +
+

It is a bad fit when:

+
    +
  • open latency is user-visible and frequent
    • +
  • the hot set is not stable
    • +
+

Constraints

+

Current hot-set warming is intentionally narrow:

+
    +
  • supported only for IncrementalDurable
    • +
  • rejected for snapshot mode
    • +
  • rejected for bounded caches and custom page-cache factories
    • +
+

Scenario 9: In-Memory Database As A Performance Tool

+

This is the right answer when durability is optional during the hot path and you can snapshot explicitly.

+

What to do

+
    +
  • Use OpenInMemoryAsync() for new ephemeral stores.
    • +
  • Use LoadIntoMemoryAsync() when you want to import a file, work in memory, then save back later.
    • +
  • Use SaveToFileAsync() at explicit persistence boundaries.
    • +
+ 
await using var db = await Database.OpenInMemoryAsync();
+                
+                // hot path
+                await db.ExecuteAsync("INSERT INTO cache VALUES (1, 'hot')");
+                
+                // persistence boundary
+                await db.SaveToFileAsync("cache.db");
+
+

Why

+

In-memory mode is the strongest write-throughput lever in the repo:

+
    +
  • hot steady-state in-memory SQL single insert: roughly 315K ops/sec
    • +
  • hot steady-state in-memory collection single put: roughly 294K ops/sec
    • +
  • in-memory batched SQL rows/sec is far above file-backed durable mode in the current macro suite
    • +
+

Use it when the workflow is:

+
    +
  • cache
    • +
  • local-first scratch workspace
    • +
  • import, process, export
    • +
  • periodically checkpointed embedded store
    • +
+

Do not use it when every commit must already be durable before control returns.

+

Scenario 10: Collection And Document Workloads

+

The collection API can be very fast, but only if you use the indexed surfaces when the workload needs them.

+

What to do

+
    +
  • Use GetAsync for key lookups.
    • +
  • Use EnsureIndexAsync(...) for fields or paths that you query repeatedly.
    • +
  • Use FindByIndexAsync(...), FindByPathAsync(...), and FindByPathRangeAsync(...) for indexed access.
    • +
  • Keep the number of indexes honest; every extra index adds write maintenance cost.
    • +
+ 
var users = await db.GetCollectionAsync<User>("users");
+                
+                await users.EnsureIndexAsync(x => x.Email);
+                await users.EnsureIndexAsync("$.address.city");
+                await users.EnsureIndexAsync("$.tags[]");
+                
+                await foreach (var match in users.FindByPathAsync("$.address.city", "Seattle"))
+                {
+                }
+
+

Why

+

Current indexed collection path numbers are strong:

+
    +
  • FindByPath("$.address.city"): about 568.9 ns
    • +
  • FindByPath("$.tags[]"): about 474.8 ns
    • +
  • FindByPath("$.orders[].sku"): about 599.3 ns
    • +
  • integer path range with 1024 matches: about 552.79 us
    • +
  • text path range with 1000 matches: about 545.69 us
    • +
+

But writes do pay for indexes:

+
    +
  • PutAsync with secondary indexes, insert case: about 12.453 us
    • +
  • PutAsync with secondary indexes, update case: about 29.079 us
    • +
  • DeleteAsync with secondary indexes: about 47.314 us
    • +
+

Practical rule

+

FindAsync(predicate) is convenient, but it is still a scan. + If the predicate matters to latency, promote it to an indexed field or indexed path.

+

Scenario 11: ADO.NET-Heavy Apps

+

This is the integration scenario: existing .NET data-access code, ORMs, utilities, or apps that open and close connections frequently.

+

What to do

+
    +
  • Turn pooling on explicitly if you open and close connections often.
    • +
  • Use private :memory: when you do not need cross-connection sharing.
    • +
  • Use named shared :memory:name only when you actually need multiple live connections against the same in-process memory database.
    • +
  • Reuse command objects when it keeps your application code cleaner, but do not expect command preparation alone to be your biggest win.
    • +
+ 
Data Source=myapp.db;Pooling=true;Max Pool Size=16
+
+ 
Data Source=:memory:
+
+ 
Data Source=:memory:shared-cache
+
+

Why

+

The current ADO.NET benchmark results are clear:

+
    +
  • open+close with pooling off: about 8.9 ms
    • +
  • open+close with pooling on: about 1.9 us
    • +
+

That is the main provider-level lever.

+

For in-memory ADO.NET:

+
    +
  • ExecuteScalar on private :memory:: 238.7 ns
    • +
  • ExecuteScalar on named shared :memory:name: 347.1 ns
    • +
  • insert on private :memory:: 2.722 us
    • +
  • insert on named shared :memory:name: 2.903 us
    • +
+

Prepared commands were roughly neutral in the current provider microbenchmarks:

+
    +
  • parameterized select: about 298.1 us
    • +
  • prepared select with reused command: about 296.7 us
    • +
+

So for most ADO.NET apps:

+
    +
  • pooling matters a lot
    • +
  • memory mode choice matters a little
    • +
  • command preparation is not the first lever to chase
    • +
+

What Usually Hurts

+

These are the common ways to leave performance on the floor in CSharpDB:

+
    +
  • Auto-committing every durable row when batching is possible.
    • +
  • Recreating a ReaderSession for every small read.
    • +
  • Using SELECT * on range and ordered queries that could be covered or at least narrow.
    • +
  • Skipping ANALYZE after bulk loads, then blaming the planner.
    • +
  • Using FindAsync(...) on large collections for fields that should be indexed.
    • +
  • Turning on advanced knobs like caching indexes, durable batch windows, or WAL preallocation without measuring the exact scenario they are meant for.
    • +
  • Paying hybrid hot-set open cost for short-lived processes.
    • +
  • Using named shared :memory: when private :memory: would do.
    • +
+ +

When tuning a real CSharpDB workload, use this order:

+
    +
  1. Pick the storage mode: file-backed, hybrid, or in-memory.
    2. +
  2. Pick the API surface: SQL, collection API, or ADO.NET.
    3. +
  3. Add the right indexes.
    4. +
  4. Shape queries so they stay narrow and covered when possible.
    5. +
  5. Run ANALYZE.
    6. +
  6. Reuse ReaderSession or pooled connections where appropriate.
    7. +
  7. Batch writes.
    8. +
  8. Only then benchmark advanced knobs.
    9. +
+

Source Map

+

Primary references for this guide:

+ +
+
+
+ + + + + diff --git a/www/docs/performance.html b/www/docs/performance.html index 9006b423..6915be94 100644 --- a/www/docs/performance.html +++ b/www/docs/performance.html @@ -96,6 +96,9 @@

On This Page

Performance Guide

Getting real performance out of CSharpDB, not just turning knobs. Organized by workload scenario with functional code examples grounded in the current benchmark suite.

+
+ Need the full source guide? The original long-form markdown version is preserved as Performance Guide Source Reference. +
Benchmark basis. Numbers are from the latest passing guardrail run (184/184 checks) on an Intel Core i9-11900K, Windows 11, .NET 10, NVMe SSD. Treat them as directional — rerun on your hardware before locking in a tuning choice. diff --git a/www/docs/pipelines.html b/www/docs/pipelines.html index 1fb41910..d117e2c7 100644 --- a/www/docs/pipelines.html +++ b/www/docs/pipelines.html @@ -40,6 +40,9 @@

ETL Pipelines

Built-in pipeline runtime for Extract-Transform-Load workflows. Define packages in JSON, validate them, serialize them, execute them in batches, and inspect run metrics.

+
+ Important naming split. This page covers product ETL pipelines. The original markdown document that had been loosely matched here described the SQL execution engine instead, and that full migrated content now lives at Query Execution Pipeline Source Reference. +

Overview

The pipeline system connects sources to destinations through a chain of transforms. Pipelines are defined as JSON package files and executed by the PipelineOrchestrator. The built-in runtime supports CSV and JSON file connectors.

diff --git a/www/docs/query-execution-pipeline.html b/www/docs/query-execution-pipeline.html new file mode 100644 index 00000000..d45cc208 --- /dev/null +++ b/www/docs/query-execution-pipeline.html @@ -0,0 +1,679 @@ + + + + + + + + Query Execution Pipeline Source Reference — CSharpDB + + + + + + + + + + + + + + +
+
+ + +
+
Source reference. This page preserves the original long-form markdown content that previously lived at docs/query-execution-pipeline.md. For the shorter curated page, see ETL Pipelines.
+

Query Execution Pipeline

+

This document describes how CSharpDB processes a SQL query from text to results. It is + intended for contributors and advanced users who want to understand how the engine makes + planning decisions.

+
+

Pipeline Overview

+ 
SQL Text
+                  │
+                  ▼
+                ┌──────────────┐
+                │  Tokenizer   │  CSharpDB.Sql/Tokenizer.cs
+                └──────┬───────┘
+                       │ Token stream
+                       ▼
+                ┌──────────────┐
+                │   Parser     │  CSharpDB.Sql/Parser.cs
+                └──────┬───────┘
+                       │ Abstract Syntax Tree (AST)
+                       ▼
+                ┌──────────────┐
+                │Query Planner │  CSharpDB.Execution/QueryPlanner.cs
+                │              │  CSharpDB.Execution/CardinalityEstimator.cs
+                └──────┬───────┘
+                       │ Physical operator tree
+                       ▼
+                ┌──────────────┐
+                │  Execution   │  CSharpDB.Execution/Operators.cs
+                └──────┬───────┘
+                       │ Row-at-a-time or batched iteration
+                       ▼
+                   Query Result
+
+
+

1. Tokenization

+

The tokenizer (Tokenizer.cs) performs a single pass over the SQL string, producing a + stream of Token(TokenType, string value, int position) objects.

+
    +
  • 120+ token types covering keywords, literals, operators, identifiers, and punctuation
    • +
  • Case-insensitive keyword matching via dictionary lookup
    • +
  • Handles SQL comments (-- ...), string escaping (''), and parameter tokens (@name)
    • +
  • Classifies numeric literals as IntegerLiteral or RealLiteral
    • +
+
+

2. Parsing

+

The parser (Parser.cs) is a recursive-descent parser that consumes tokens and builds a + typed AST.

+

AST Node Categories

+

Statements — top-level SQL commands:

+
    +
  • DDL: CreateTableStatement, AlterTableStatement, DropTableStatement, + CreateIndexStatement, DropIndexStatement, CreateViewStatement, DropViewStatement, + CreateTriggerStatement, DropTriggerStatement, AnalyzeStatement
    • +
  • DML: InsertStatement, UpdateStatement, DeleteStatement
    • +
  • Query: SelectStatement, CompoundSelectStatement (set operations), WithStatement (CTEs)
    • +
+

Table references:

+
    +
  • SimpleTableRef(TableName, Alias) — single table or view
    • +
  • JoinTableRef(Left, Right, JoinType, Condition) — binary join tree
    • +
+

Expressions:

+
    +
  • LiteralExpression, ParameterExpression, ColumnRefExpression
    • +
  • BinaryExpression — arithmetic, comparison, logical operators
    • +
  • UnaryExpression — NOT, unary minus
    • +
  • FunctionCallExpression — aggregates and scalar functions
    • +
  • LikeExpression, InExpression, BetweenExpression, IsNullExpression
    • +
  • ScalarSubqueryExpression, InSubqueryExpression, ExistsExpression
    • +
  • CollateExpression
    • +
+

Parser Fast Paths

+

The parser detects common query shapes and produces lightweight metadata instead of a full + AST, skipping unnecessary allocation:

+ + + + + + + + + + + + + + + + + + + + + + + + + +
Fast PathPattern DetectedBenefit
Simple SELECTSingle table, simple WHERESkips full AST construction
Primary key lookupSELECT ... WHERE pk = valueDirect lookup metadata
Simple INSERTINSERT INTO ... VALUES (...)Lightweight insert path
+
+

3. Query Planning

+

The QueryPlanner dispatches statements to type-specific handlers. For SELECT queries, + it classifies the query into a SelectPlanKind to choose an execution strategy:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Plan KindDescription
FastPrimaryKeyLookupDirect integer PK lookup — fastest path
FastIndexedLookupEquality lookup via index
FastSimpleTableScanFull scan with basic filtering
SimpleCountStarCOUNT(*) using cached row count (no scan)
SimpleScalarAggregateColumnSingle aggregate on a column
SimpleGroupedIndexAggregateGROUP BY on indexed column (streaming)
GeneralFull operator tree for complex queries
+

Plan classifications are cached (up to 1024 entries) so repeated queries skip re-analysis.

+

Cardinality Estimation

+

When ANALYZE has been run on a table, the CardinalityEstimator uses collected statistics + to estimate result sizes and guide operator selection.

+

Statistics collected by ANALYZE:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StatisticScopeDescription
Distinct countPer columnNumber of unique values
Non-NULL countPer columnNumber of non-NULL values
Min / Max valuePer columnValue range
Frequent valuesPer columnTop 8 values by occurrence
Histogram bucketsPer column16 quantile-based buckets for range estimation
Prefix distinct countsPer indexDistinct values for each prefix of a composite index
+

Estimation methods:

+
    +
  • Lookup selectivity: tableRows / distinctCount for uniform distribution, adjusted + for skew when the lookup value matches a known frequent value
    • +
  • Filter selectivity: Per-column selectivity multiplied across AND-conjuncts. Supports + equality, range, NULL, and discrete value list predicates.
    • +
  • Join cardinality: (leftRows × rightRows) / max(leftDistinct, rightDistinct), + adjusted by non-NULL fraction and outer join semantics.
    • +
+

Without statistics, the planner uses heuristic fallback estimates.

+

Index Selection

+

For each AND-separated predicate in the WHERE clause, the planner checks whether an + available index can satisfy it:

+ + + + + + + + + + + + + + + + + + + + + + + + + +
CandidateRankDescription
Integer primary key equality0Best — direct B-tree lookup
Unique index equality1Single-row guarantee
Non-unique index equality2Accepted only if estimated rows ≤ 25% of table
+

When multiple candidates match, the planner picks the lowest rank. Ties are broken by + estimated row count (fewer is better). Composite indexes are used when equality predicates + match the index prefix columns.

+

Join Operator Selection

+

The planner tries join operators in preference order:

+

1. Index Nested-Loop Join

+

Requirements: INNER or LEFT join, right side is a simple table, equi-join condition exists, + and the right table has an index on the join key.

+

Cost decision (when statistics are available):

+
    +
  • Unique index: use if leftRows ≤ rightRows × 8 (PK) or leftRows ≤ rightRows × 2 + (unique)
    • +
  • Non-unique index: use if estimated lookup cost < hash join cost
    • +
+

Produces IndexNestedLoopJoinOperator (integer PK) or HashedIndexNestedLoopJoinOperator + (text/composite keys).

+

2. Hash Join

+

Requirements: equi-join condition with extractable key columns.

+

Build side selection: for INNER joins, builds the smaller estimated side. For LEFT/RIGHT + joins, respects outer table preservation.

+

3. Nested-Loop Join (fallback)

+

Cartesian product with join condition filtering. Used when no index or equi-join condition + is available.

+

Join Reordering

+

For queries joining 4–6 tables, the planner applies dynamic-programming-based join + reordering to find the lowest-cardinality ordering. Results are cached per query shape. + Views, CTEs, and system tables are excluded from reordering.

+
+

4. Execution

+

Iterator Model

+

Operators implement a pull-based iterator protocol:

+ 
interface IOperator : IAsyncDisposable
+                {
+                    ColumnDefinition[] OutputSchema { get; }
+                    ValueTask OpenAsync();
+                    ValueTask<bool> MoveNextAsync();
+                    DbValue[] Current { get; }
+                }
+
+

The root operator is opened, then MoveNextAsync is called repeatedly until it returns + false. Results flow from leaf operators (scans) upward through the tree.

+

Batch Execution

+

Performance-critical operators also implement IBatchOperator, which yields RowBatch + objects containing up to 64 rows per batch. This enables SIMD-friendly memory layout and + reduces per-row overhead.

+ 
interface IBatchOperator : IAsyncDisposable
+                {
+                    ValueTask OpenAsync();
+                    ValueTask<bool> MoveNextBatchAsync();
+                    RowBatch CurrentBatch { get; }
+                }
+
+

Expression Evaluation

+

Expressions are evaluated through two paths:

+ + + + + + + + + + + + + + + + + + + + +
PathUsed WhenHow
InterpreterOne-off evaluationRecursive AST walk with per-row schema lookups
CompiledHot paths (filters, projections)AST compiled to Func<DbValue[], DbValue> delegate with bound column indices. Cached (up to 4096 entries).
+

Operator Catalog

+

Scan Operators

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorDescription
TableScanOperatorSequential B+tree scan. Supports pre-decode filtering and projection pushdown.
IndexScanOperatorOrdered index scan with optional range bounds.
IndexOrderedScanOperatorPrefix scan with secondary range filtering.
PrimaryKeyLookupOperatorDirect row lookup by integer PK. Fastest scan path.
UniqueIndexLookupOperatorSingle-row lookup via unique index.
+

Projection variants (PrimaryKeyProjectionLookupOperator, + IndexScanProjectionOperator, etc.) fuse column selection into the scan to skip + unnecessary deserialization.

+

Join Operators

+ + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorDescription
HashJoinOperatorHash table on build side, probed by probe side. Supports INNER, LEFT, RIGHT, CROSS.
IndexNestedLoopJoinOperatorProbes right table via integer PK index for each left row.
HashedIndexNestedLoopJoinOperatorProbes via hash-based index (text/composite keys).
NestedLoopJoinOperatorCartesian product with condition filtering.
+

Filter and Projection

+ + + + + + + + + + + + + + + + + + + + + +
OperatorDescription
FilterOperatorApplies WHERE predicates. Optional compiled filter for hot paths.
ProjectionOperatorColumn selection and expression evaluation.
FilterProjectionOperatorFused filter + projection in a single pass.
+

Aggregation

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorDescription
HashAggregateOperatorGrouped aggregation via hash table (GROUP BY).
ScalarAggregateOperatorSingle-row aggregation (no GROUP BY).
IndexGroupedAggregateOperatorStreaming GROUP BY on indexed column (no hash table).
CompositeIndexGroupedAggregateOperatorStreaming GROUP BY on composite index prefix.
CountStarTableOperatorReturns cached row count — no scan needed.
+

Additional specialized variants exist for specific aggregate patterns + (ScalarAggregateLookupOperator, FilteredScalarAggregateTableOperator, etc.).

+

Sort and Limit

+ + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorDescription
SortOperatorFull sort (in-memory for small sets, merge sort for larger).
TopNSortOperatorHeap-based top-N for ORDER BY ... LIMIT N — avoids full sort.
LimitOperatorTruncates output after N rows.
OffsetOperatorSkips first N rows.
+

Other

+ + + + + + + + + + + + + + + + + +
OperatorDescription
DistinctOperatorRemoves duplicates via hash set.
MaterializedOperatorWraps pre-materialized rows (CTEs, subquery results).
+

Optimization Interfaces

+

Operators expose capability interfaces that enable cross-operator optimization:

+ + + + + + + + + + + + + + + + + + + + + + + + + +
InterfacePurpose
IPreDecodeFilterSupportPush equality filters into the decode layer, eliminating rows before full deserialization
IProjectionPushdownTargetDeclare needed output columns so scans skip unused columns
IEstimatedRowCountProviderExpose cardinality to parent operators for cost decisions
IRowBufferReuseControllerControl whether row buffers are reused or cloned
+
+

5. Worked Example

+ 
SELECT c.name, SUM(o.amount) AS total
+                FROM customers c
+                LEFT JOIN orders o ON o.cust_id = c.id
+                WHERE c.country = 'USA'
+                GROUP BY c.id, c.name
+                ORDER BY total DESC
+                LIMIT 10;
+
+

Planning decisions:

+
    +
  1. +

    Index check on customers.country: Non-unique index found. Estimated 300K rows out + of 1M — selectivity passes the 25% threshold → use IndexScanOperator.

    +
    2. +
  2. +

    Join strategy: Index exists on orders.cust_id. Estimated left rows (300K) ≤ + right rows × 8 → use IndexNestedLoopJoinOperator.

    +
    3. +
  3. +

    Aggregation: GROUP BY columns not aligned with an index → HashAggregateOperator.

    +
    4. +
  4. +

    Sort + limit: ORDER BY total DESC LIMIT 10 → TopNSortOperator (heap of size 10, + avoids sorting all groups).

    +
    5. +
+

Resulting operator tree:

+ 
LimitOperator (10)
+                  └─ TopNSortOperator (total DESC, N=10)
+                       └─ HashAggregateOperator (GROUP BY c.id, c.name; SUM(o.amount))
+                            └─ IndexNestedLoopJoinOperator (LEFT, o.cust_id = c.id)
+                                 ├─ IndexScanOperator (customers, country = 'USA')
+                                 └─ orders.cust_id index
+
+

Execution:

+
    +
  1. IndexScanOperator iterates the country index for 'USA' entries, yielding customer + rows in batches of 64.
    2. +
  2. For each customer, IndexNestedLoopJoinOperator probes the orders index on cust_id, + yielding joined rows (or a NULL-padded row for LEFT JOIN when no orders exist).
    3. +
  3. HashAggregateOperator accumulates SUM(amount) per (id, name) group in a hash + table, then emits all groups.
    4. +
  4. TopNSortOperator maintains a min-heap of 10 entries by total DESC, discarding rows + below the current 10th largest.
    5. +
  5. LimitOperator yields the final 10 rows to the caller.
    6. +
+
+

Constants

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ConstantValuePurpose
Default batch size64 rowsRowBatch capacity
Expression cache4096 entriesCompiled expression delegate cache
Plan cache1024 entriesSELECT plan classification cache
Histogram buckets16Quantile buckets per column in ANALYZE
Frequent values8Top-N tracked per column in ANALYZE
Max join reorder leaves6DP-based reordering table limit
Max trigger depth16Recursive trigger invocation limit
Max FK cascade depth64Foreign key cascade recursion limit
+
+
+
+ + + + + diff --git a/www/docs/sql-reference.html b/www/docs/sql-reference.html new file mode 100644 index 00000000..54bd2753 --- /dev/null +++ b/www/docs/sql-reference.html @@ -0,0 +1,601 @@ + + + + + + + + SQL Source Reference — CSharpDB + + + + + + + + + + + + + + +
+
+ + +
+
Source reference. This page preserves the original long-form markdown content that previously lived at docs/sql-reference.md. For the shorter curated page, see SQL Reference.
+

SQL Reference

+

Complete reference for the SQL dialect supported by CSharpDB.

+
+

Data Types

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeAliasesDescription
INTEGERINT64-bit signed integer
REALFLOAT, DOUBLE64-bit IEEE 754 floating point
TEXTVARCHARUTF-8 Unicode string
BLOB—Raw binary data
NULL—Explicit NULL value (any column unless constrained NOT NULL)
+

CSharpDB uses a flexible type system. Arithmetic operators perform implicit coercion + between numeric types where needed.

+
+

Statements

+

CREATE TABLE

+ 
CREATE TABLE [IF NOT EXISTS] table_name (
+                    column_name type [PRIMARY KEY] [IDENTITY | AUTOINCREMENT] [NOT NULL]
+                                     [COLLATE collation_name]
+                                     [REFERENCES other_table(column) [ON DELETE CASCADE | RESTRICT]],
+                    ...
+                );
+
+

Constraints:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ConstraintScopeDescription
PRIMARY KEYColumnDesignates the row identity column (integer)
IDENTITY / AUTOINCREMENTColumnAuto-incrementing integer primary key
NOT NULLColumnRejects NULL values on insert/update
COLLATEColumnSets collation for TEXT comparisons (see Collations)
REFERENCESColumnForeign key referencing another table's column
ON DELETE CASCADEForeign keyDeletes child rows when parent is deleted
ON DELETE RESTRICTForeign keyPrevents deletion of parent row while children exist
+

ALTER TABLE

+ 
ALTER TABLE table_name ADD COLUMN column_name type [constraints];
+                ALTER TABLE table_name DROP COLUMN column_name;
+                ALTER TABLE table_name DROP CONSTRAINT constraint_name;
+                ALTER TABLE table_name RENAME TO new_name;
+                ALTER TABLE table_name RENAME COLUMN old_name TO new_name;
+
+

DROP TABLE

+ 
DROP TABLE [IF EXISTS] table_name;
+
+

CREATE INDEX

+ 
CREATE [UNIQUE] INDEX [IF NOT EXISTS] index_name
+                ON table_name (column1 [, column2, ...]);
+
+

DROP INDEX

+ 
DROP INDEX [IF EXISTS] index_name;
+
+

CREATE VIEW

+ 
CREATE VIEW [IF NOT EXISTS] view_name AS select_statement;
+
+

DROP VIEW

+ 
DROP VIEW [IF EXISTS] view_name;
+
+

CREATE TRIGGER

+ 
CREATE TRIGGER [IF NOT EXISTS] trigger_name
+                {BEFORE | AFTER} {INSERT | UPDATE | DELETE}
+                ON table_name
+                [FOR EACH ROW]
+                [WHEN condition]
+                BEGIN
+                    statement1;
+                    [statement2;]
+                    ...
+                END;
+
+

Triggers can reference NEW and OLD row aliases in their body and WHEN condition:

+
    +
  • INSERT triggers: NEW is available
    • +
  • DELETE triggers: OLD is available
    • +
  • UPDATE triggers: both NEW and OLD are available
    • +
+

DROP TRIGGER

+ 
DROP TRIGGER [IF EXISTS] trigger_name;
+
+

ANALYZE

+ 
ANALYZE table_name;
+
+

Collects per-column statistics (distinct count, min/max, frequency histograms, quantile + buckets) and index prefix statistics used by the query planner for cardinality estimation + and operator selection. See Query Execution Pipeline for + details on how statistics influence planning.

+
+

Data Manipulation

+

INSERT

+ 
INSERT INTO table_name [(column1, column2, ...)]
+                VALUES (value1, value2, ...);
+
+

Column list is optional when providing values for all columns in declaration order.

+

UPDATE

+ 
UPDATE table_name
+                SET column1 = expression1 [, column2 = expression2, ...]
+                [WHERE condition];
+
+

DELETE

+ 
DELETE FROM table_name
+                [WHERE condition];
+
+

SELECT

+ 
SELECT [DISTINCT] column_list
+                FROM table_reference
+                [JOIN ...]
+                [WHERE condition]
+                [GROUP BY column1 [, column2, ...]]
+                [HAVING condition]
+                [ORDER BY column1 [ASC | DESC] [, ...]]
+                [LIMIT count]
+                [OFFSET skip];
+
+

Column List

+ 
SELECT *                              -- all columns
+                SELECT column_name                    -- single column
+                SELECT column_name AS alias           -- aliased column
+                SELECT table.column_name              -- qualified column
+                SELECT expression                     -- computed value
+                SELECT aggregate_function(...)        -- aggregate
+
+

FROM and JOIN

+ 
FROM table_name [AS alias]
+                
+                -- Join types
+                INNER JOIN table_name ON condition
+                LEFT  JOIN table_name ON condition
+                RIGHT JOIN table_name ON condition
+                CROSS JOIN table_name
+
+

All join types except CROSS JOIN require an ON condition.

+

Subqueries

+ 
-- Scalar subquery (must return a single value)
+                SELECT (SELECT MAX(age) FROM users) AS max_age;
+                
+                -- IN subquery
+                WHERE column IN (SELECT id FROM other_table)
+                WHERE column NOT IN (SELECT id FROM other_table)
+                
+                -- EXISTS subquery
+                WHERE EXISTS (SELECT 1 FROM other_table WHERE condition)
+
+
+

Common Table Expressions (CTEs)

+ 
WITH cte_name [(column1, column2, ...)] AS (
+                    select_statement
+                )
+                [, another_cte AS (...)]
+                SELECT ... FROM cte_name ...;
+
+

Multiple CTEs can be chained with commas. Optional column name lists rename the CTE's + output columns.

+
+

Note: The RECURSIVE keyword is parsed but recursive CTE execution is not yet + implemented.

+
+
+

Set Operations

+ 
select_statement UNION     select_statement
+                select_statement INTERSECT select_statement
+                select_statement EXCEPT    select_statement
+
+

Compound queries support trailing ORDER BY, LIMIT, and OFFSET applied to the + combined result.

+
+

Expressions and Operators

+

Arithmetic

+ + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorDescription
+Addition
-Subtraction (also unary negation)
*Multiplication
/Division (error on division by zero)
+

Comparison

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperatorDescription
=Equal
<> or !=Not equal
<Less than
>Greater than
<=Less than or equal
>=Greater than or equal
+

Logical

+ + + + + + + + + + + + + + + + + + + + + +
OperatorDescription
ANDLogical conjunction
ORLogical disjunction
NOTLogical negation
+

Special Expressions

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ExpressionExample
BETWEEN ... AND ...WHERE age BETWEEN 18 AND 65
IN (...)WHERE status IN ('active', 'pending')
NOT IN (...)WHERE id NOT IN (1, 2, 3)
LIKEWHERE name LIKE 'J%'
LIKE ... ESCAPEWHERE code LIKE '100\%%' ESCAPE '\'
IS NULLWHERE email IS NULL
IS NOT NULLWHERE email IS NOT NULL
+

LIKE wildcards:

+ + + + + + + + + + + + + + + + + +
WildcardMatches
%Zero or more characters
_Exactly one character
+
+

Functions

+

Aggregate Functions

+

Used with or without GROUP BY. All except COUNT(*) ignore NULL values.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FunctionDescriptionSupports DISTINCT
COUNT(*)Number of rows—
COUNT(expr)Number of non-NULL valuesYes
SUM(expr)Sum of numeric valuesYes
AVG(expr)Average of numeric valuesYes
MIN(expr)Minimum value—
MAX(expr)Maximum value—
+ 
SELECT COUNT(DISTINCT status), AVG(age) FROM users;
+
+

Scalar Functions

+ + + + + + + + + + + + + + + + + +
FunctionArgumentsReturnsDescription
TEXT(expr)1TEXTConverts any value to its text representation
+
+

Parameters

+

Named parameters are supported in value positions using the @ prefix:

+ 
SELECT * FROM users WHERE name = @name AND age > @minAge;
+                INSERT INTO users (name, age) VALUES (@name, @age);
+                UPDATE users SET name = @name WHERE id = @id;
+                DELETE FROM users WHERE id = @id;
+
+

Parameters cannot be used in identifier positions (table names, column names).

+
+

Collations

+

Collations control how TEXT values are compared and sorted. They can be specified at the + column level in CREATE TABLE or at the expression level using the COLLATE operator.

+ + + + + + + + + + + + + + + + + + + + + + + + + +
CollationDescription
BINARYByte-for-byte comparison (default)
NOCASECase-insensitive comparison
NOCASE_AICase-insensitive and accent-insensitive comparison
ICU:<locale>Unicode ICU-based comparison with locale support
+ 
-- Column-level collation
+                CREATE TABLE products (
+                    name TEXT COLLATE NOCASE
+                );
+                
+                -- Expression-level collation
+                SELECT * FROM products ORDER BY name COLLATE NOCASE_AI;
+
+
+

Limitations

+

The following SQL features are not currently supported:

+
    +
  • CASE / WHEN expressions
    • +
  • CAST expressions (implicit coercion only)
    • +
  • DEFAULT column values
    • +
  • CHECK constraints
    • +
  • RETURNING clause on INSERT/UPDATE/DELETE
    • +
  • UPSERT / ON CONFLICT / INSERT OR REPLACE
    • +
  • Recursive CTE execution (WITH RECURSIVE is parsed but not evaluated)
    • +
  • String functions (UPPER, LOWER, LENGTH, SUBSTR, TRIM)
    • +
  • Date/time functions (DATE, TIME, DATETIME, STRFTIME)
    • +
  • Math functions (ABS, ROUND, CEIL, FLOOR)
    • +
  • Window functions (OVER, PARTITION BY, ROW_NUMBER, etc.)
    • +
  • Stored procedures
    • +
  • Composite primary keys / composite foreign keys
    • +
+
+
+
+ + + + + diff --git a/www/docs/sql.html b/www/docs/sql.html index 6b14f9ce..7ce25efb 100644 --- a/www/docs/sql.html +++ b/www/docs/sql.html @@ -46,6 +46,9 @@

SQL Reference

CSharpDB includes a full SQL engine with tokenizer, parser, query planner, and expression evaluator.

+
+ Need the full source guide? The original long-form markdown version is preserved as SQL Source Reference. +

Data Definition (DDL)

diff --git a/www/docs/storage-engine-reference.html b/www/docs/storage-engine-reference.html new file mode 100644 index 00000000..acf3752e --- /dev/null +++ b/www/docs/storage-engine-reference.html @@ -0,0 +1,1388 @@ + + + + + + + + Storage Engine Source Reference — CSharpDB + + + + + + + + + + + + + + +
+
+ + +
+
Source reference. This page preserves the original long-form markdown content that previously lived at docs/storage/README.md. For the shorter curated page, see Storage Engine Reference.
+

CSharpDB.Storage

+

A low-level, high-performance storage engine for .NET 10 built on top of RandomAccess and SafeFileHandle. It provides random-access async I/O, page caching, write-ahead logging (WAL), crash recovery, and the B+tree/index primitives that power the SQL engine and collection API.

+

For the guided storage tutorial track, including the new advanced standalone examples, start with samples/storage-tutorials/README.md. For the package-oriented storage API overview and tuning presets, see src/CSharpDB.Storage/README.md.

+
+

Table of Contents

+ +
+

Architecture Overview

+ 
┌──────────────────────────────────────────────────────┐
+                │                   Application                        │
+                │          (SQL Engine / Collection API)               │
+                ├──────────────────────────────────────────────────────┤
+                │                 SchemaCatalog                        │
+                │    Tables ─ Indexes ─ Views ─ Triggers               │
+                ├──────────────┬───────────────┬───────────────────────┤
+                │    BTree     │  IndexStore   │  RecordEncoder        │
+                │  (data)      │  (secondary)  │  (row format)         │
+                ├──────────────┴───────────────┴───────────────────────┤
+                │                    Pager                             │
+                │   PageCache ─ DirtyTracking ─ PageAllocator          │
+                ├──────────────────────┬───────────────────────────────┤
+                │   WriteAheadLog      │   CheckpointCoordinator       │
+                │   (WAL + WalIndex)   │   (policy-driven)             │
+                ├──────────────────────┴───────────────────────────────┤
+                │              IStorageDevice                          │
+                │         (FileStorageDevice / memory)                 │
+                └──────────────────────────────────────────────────────┘
+
+

Current storage builds can also enable a few newer read-path behaviors on top of this core layout:

+
    +
  • Memory-mapped reads for clean main-file pages when the storage device supports it
    • +
  • Speculative B+tree leaf read-ahead during sequential forward scans
    • +
  • Checkpoint residency preservation so already-owned main-file pages stay hot across checkpoint in lazy-resident hybrid engine mode
    • +
+

Page Layout:

+ 
Page 0 (File Header):
+                  [Magic: 4 bytes "CSDB"]
+                  [FormatVersion: 4 bytes]
+                  [PageSize: 4 bytes = 4096]
+                  [PageCount: 4 bytes]
+                  [SchemaRootPage: 4 bytes]
+                  [FreelistHead: 4 bytes]
+                  [ChangeCounter: 4 bytes]
+                  [... reserved to 100 bytes ...]
+                  [Slotted page content: 3996 bytes]
+                
+                Pages 1+:
+                  [SlottedPage: 4096 bytes]
+                    [Header: 9 bytes]
+                      PageType (1) ─ CellCount (2) ─ CellContentStart (2) ─ RightChild/NextLeaf (4)
+                    [CellPointers: 2 bytes each]
+                    [Free space]
+                    [Cells: growing backward from page end]
+
+
+

FileStorageDevice

+

FileStorageDevice wraps a SafeFileHandle opened with FileOptions.Asynchronous | FileOptions.RandomAccess, giving you:

+
    +
  • True async I/O via RandomAccess.ReadAsync / RandomAccess.WriteAsync
    • +
  • Position-independent reads and writes -- no seek, no shared file pointer
    • +
  • Concurrent reads from other processes (FileShare.Read)
    • +
  • Direct fsync to durable storage via RandomAccess.FlushToDisk
    • +
+ 
public FileStorageDevice(string filePath, bool createNew = false)
+
+ + + + + + + + + + + + + + + + + +
ParameterDescription
filePathPath to the database file.
createNewtrue -> FileMode.CreateNew (fails if file exists). false -> OpenOrCreate.
+
+

IStorageDevice Interface

+

All storage operations go through IStorageDevice, making it easy to swap implementations (e.g., for in-memory testing).

+ 
public interface IStorageDevice : IAsyncDisposable, IDisposable
+                {
+                    long Length { get; }
+                    ValueTask<int> ReadAsync(long offset, Memory<byte> buffer, CancellationToken ct = default);
+                    ValueTask WriteAsync(long offset, ReadOnlyMemory<byte> buffer, CancellationToken ct = default);
+                    ValueTask FlushAsync(CancellationToken ct = default);
+                    ValueTask SetLengthAsync(long length, CancellationToken ct = default);
+                }
+
+
+

Device Scenarios

+

1. Create a New File

+

Creates the file, failing if it already exists.

+ 
await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+                Console.WriteLine($"File created. Length: {device.Length}"); // 0
+
+

2. Open an Existing File

+

Opens the file if it exists, or creates it if it does not.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                Console.WriteLine($"Opened. Length: {device.Length}");
+
+

3. Write Raw Bytes at an Offset

+

Writes are position-independent. Multiple writes at different offsets can be issued concurrently without locking.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                byte[] payload = "Hello, CSharpDB!"u8.ToArray();
+                await device.WriteAsync(offset: 0, payload);
+
+

4. Read Raw Bytes from an Offset

+

ReadAsync loops internally until the buffer is fully filled or EOF is reached.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                var buffer = new byte[16];
+                int bytesRead = await device.ReadAsync(offset: 0, buffer);
+                Console.WriteLine($"Read {bytesRead} byte(s): {System.Text.Encoding.UTF8.GetString(buffer, 0, bytesRead)}");
+
+

5. Read Past End of File (Zero-Fill Behavior)

+

If you read a range that extends beyond the current file length, ReadAsync zero-fills the remainder of the buffer and returns the number of bytes that were actually on disk. This is useful for treating uninitialized pages as zeroed memory.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                // File is currently 16 bytes; request 4096 bytes
+                var buffer = new byte[4096];
+                int bytesRead = await device.ReadAsync(offset: 0, buffer);
+                Console.WriteLine($"Bytes on disk: {bytesRead}");          // 16
+                Console.WriteLine($"Remainder is zeros: {buffer[16] == 0}"); // true
+
+

6. Pre-allocate / Extend File Length

+

Pre-allocating avoids fragmentation on spinning disks and is required before writing pages beyond the current end-of-file on some file systems.

+ 
await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+                const int PageSize = 4096;
+                const int InitialPages = 8;
+                await device.SetLengthAsync(PageSize * InitialPages);
+                Console.WriteLine($"Pre-allocated: {device.Length} bytes"); // 32768
+
+

7. Flush to Disk (fsync)

+

FlushAsync calls RandomAccess.FlushToDisk, which issues a full fsync/FlushFileBuffers. Call this after committing a transaction to guarantee durability.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                byte[] data = new byte[4096];
+                await device.WriteAsync(offset: 0, data);
+                await device.FlushAsync(); // durable on disk after this returns
+
+

8. Check File Length

+

The Length property reads the current file size directly from the OS without seeking.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                long pages = device.Length / 4096;
+                Console.WriteLine($"Database has {pages} page(s).");
+
+

9. Dispose Synchronously

+

FileStorageDevice implements IDisposable for non-async contexts such as unit tests or top-level using statements.

+ 
using var device = new FileStorageDevice("mydb.cdb");
+                // ... operations ...
+                // Disposed when leaving the scope.
+
+

10. Dispose Asynchronously

+

Prefer await using in async code paths to align with the async I/O model.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                // ... operations ...
+                // DisposeAsync called automatically.
+
+

11. Cancellation Support

+

All async methods accept a CancellationToken. Pass one to abort long reads or writes cleanly.

+ 
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
+                await using var device = new FileStorageDevice("mydb.cdb");
+                var buffer = new byte[4096];
+                try
+                {
+                    await device.ReadAsync(offset: 0, buffer, cts.Token);
+                }
+                catch (OperationCanceledException)
+                {
+                    Console.WriteLine("Read was cancelled.");
+                }
+
+

12. Injecting via IStorageDevice (Testability)

+

Program against IStorageDevice so you can substitute a fake or in-memory implementation in tests without touching the file system.

+ 
// Production wiring
+                IStorageDevice device = new FileStorageDevice("mydb.cdb");
+                var pager = await Pager.CreateAsync(device, wal, walIndex);
+                
+                // In a unit test -- swap in your own IStorageDevice implementation
+                public sealed class MemoryStorageDevice : IStorageDevice
+                {
+                    private byte[] _data = [];
+                    public long Length => _data.Length;
+                
+                    public ValueTask<int> ReadAsync(long offset, Memory<byte> buffer, CancellationToken ct = default)
+                    {
+                        int available = (int)Math.Max(0, _data.Length - offset);
+                        int toCopy = Math.Min(buffer.Length, available);
+                        _data.AsMemory((int)offset, toCopy).CopyTo(buffer);
+                        buffer[toCopy..].Span.Clear();
+                        return ValueTask.FromResult(toCopy);
+                    }
+                
+                    public ValueTask WriteAsync(long offset, ReadOnlyMemory<byte> buffer, CancellationToken ct = default)
+                    {
+                        long needed = offset + buffer.Length;
+                        if (needed > _data.Length)
+                            Array.Resize(ref _data, (int)needed);
+                        buffer.CopyTo(_data.AsMemory((int)offset));
+                        return ValueTask.CompletedTask;
+                    }
+                
+                    public ValueTask FlushAsync(CancellationToken ct = default) => ValueTask.CompletedTask;
+                
+                    public ValueTask SetLengthAsync(long length, CancellationToken ct = default)
+                    {
+                        Array.Resize(ref _data, (int)length);
+                        return ValueTask.CompletedTask;
+                    }
+                
+                    public ValueTask DisposeAsync() => ValueTask.CompletedTask;
+                    public void Dispose() { }
+                }
+
+

13. Writing Fixed-Size Pages (4 KB)

+

The storage engine works in 4 096-byte pages (see PageConstants.PageSize). Write a page at a computed offset.

+ 
await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+                const int PageSize = 4096;
+                
+                // Pre-allocate space for 4 pages
+                await device.SetLengthAsync(PageSize * 4);
+                
+                // Build a page payload
+                byte[] page = new byte[PageSize];
+                System.Text.Encoding.UTF8.GetBytes("page-0 data").CopyTo(page.AsSpan());
+                
+                // Write page 0 at offset 0, page 1 at offset 4096, etc.
+                uint pageId = 0;
+                await device.WriteAsync(offset: (long)pageId * PageSize, page);
+
+

14. Reading Fixed-Size Pages (4 KB)

+

Read a page back by computing its byte offset from its page number.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                const int PageSize = 4096;
+                
+                uint pageId = 0;
+                byte[] page = new byte[PageSize];
+                int read = await device.ReadAsync(offset: (long)pageId * PageSize, page);
+                Console.WriteLine($"Page {pageId}: read={read}, first bytes={System.Text.Encoding.UTF8.GetString(page, 0, 10)}");
+
+

15. Appending Sequential Pages

+

Grow the file by one page at a time and write content into each new page.

+ 
await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+                const int PageSize = 4096;
+                int pagesToAppend = 3;
+                
+                for (int i = 0; i < pagesToAppend; i++)
+                {
+                    long newLength = device.Length + PageSize;
+                    await device.SetLengthAsync(newLength);
+                
+                    byte[] page = new byte[PageSize];
+                    BitConverter.TryWriteBytes(page, i); // store page index in first 4 bytes
+                    await device.WriteAsync(offset: newLength - PageSize, page);
+                }
+                
+                Console.WriteLine($"Final file size: {device.Length} bytes"); // 12288
+                await device.FlushAsync();
+
+
+

Pager

+

The Pager sits between the B+tree layer and the storage device. It owns the page cache, tracks dirty pages, coordinates transactions, manages WAL integration, and drives checkpointing.

+ 
public static async ValueTask<Pager> CreateAsync(
+                    IStorageDevice device,
+                    IWriteAheadLog wal,
+                    WalIndex walIndex,
+                    PagerOptions? options = null,
+                    CancellationToken ct = default)
+
+

P1. Create a New Database

+ 
await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+                var walIndex = new WalIndex();
+                await using var wal = new WriteAheadLog("mydb.cdb", walIndex);
+                await wal.OpenAsync(currentDbPageCount: 0);
+                
+                var pager = await Pager.CreateAsync(device, wal, walIndex);
+                await pager.InitializeNewDatabaseAsync(); // writes file header (page 0)
+                
+                Console.WriteLine($"Pages: {pager.PageCount}"); // 1 (the file header page)
+
+

P2. Open and Recover an Existing Database

+

On startup, call RecoverAsync to redo any committed WAL frames that were not yet checkpointed.

+ 
await using var device = new FileStorageDevice("mydb.cdb");
+                var walIndex = new WalIndex();
+                await using var wal = new WriteAheadLog("mydb.cdb", walIndex);
+                
+                var pager = await Pager.CreateAsync(device, wal, walIndex);
+                await pager.RecoverAsync(); // replays committed WAL frames
+
+

P3. Read and Write Pages

+ 
// Read a page (checks cache -> WAL -> device)
+                byte[] page = await pager.GetPageAsync(pageId: 1);
+                
+                // Modify the page buffer in-place, then mark dirty
+                page[0] = 0xFF;
+                await pager.MarkDirtyAsync(pageId: 1); // tracked for WAL write on commit
+
+

P4. Allocate and Free Pages

+ 
// Allocate a new page (extends file or reuses from freelist)
+                uint newPageId = await pager.AllocatePageAsync();
+                
+                // Free a page (adds to freelist for reuse)
+                await pager.FreePageAsync(newPageId);
+
+

P5. Transaction Lifecycle

+

Single writer per database. Reads do not require transactions.

+ 
await pager.BeginTransactionAsync();
+                try
+                {
+                    // ... modify pages via B+tree ...
+                    await pager.CommitAsync(); // writes dirty pages to WAL, fsync
+                }
+                catch
+                {
+                    await pager.RollbackAsync(); // discards uncommitted WAL frames
+                    throw;
+                }
+
+

P6. Snapshot Isolation (Concurrent Readers)

+

Multiple readers can run concurrently with a single writer. Each reader sees a consistent snapshot.

+ 
// Writer thread: acquire snapshot for a reader
+                WalSnapshot snapshot = pager.AcquireReaderSnapshot();
+                
+                // Reader thread: create a snapshot pager that sees only committed data
+                Pager snapshotPager = pager.CreateSnapshotReader(snapshot);
+                byte[] page = await snapshotPager.GetPageAsync(pageId: 1);
+                
+                // When reader is done:
+                pager.ReleaseReaderSnapshot();
+
+

P7. Manual Checkpoint

+

Copy all committed WAL frames to the main database file, then reset the WAL. This invalidates transient WAL-backed reads, and can also invalidate transient memory-mapped views; with PreserveOwnedPagesOnCheckpoint = true, already-owned main-file pages can remain resident in the shared cache.

+ 
await pager.CheckpointAsync();
+
+

P8. Configure Checkpoint Policy

+ 
var options = new PagerOptions
+                {
+                    CheckpointPolicy = new AnyCheckpointPolicy(
+                        new FrameCountCheckpointPolicy(threshold: 500),
+                        new TimeIntervalCheckpointPolicy(TimeSpan.FromMinutes(5))
+                    ),
+                    AutoCheckpointExecutionMode = AutoCheckpointExecutionMode.Background,
+                    AutoCheckpointMaxPagesPerStep = 64
+                };
+                
+                var pager = await Pager.CreateAsync(device, wal, walIndex, options);
+                // Auto-checkpoint triggers after 500 frames OR 5 minutes, whichever comes first.
+                // In background mode, the checkpoint runs after commit in smaller slices.
+
+

Built-in policies:

+ + + + + + + + + + + + + + + + + + + + + + + + + +
PolicyTriggers When
FrameCountCheckpointPolicy(n)Committed frame count exceeds n
WalSizeCheckpointPolicy(bytes)Estimated WAL size exceeds bytes
TimeIntervalCheckpointPolicy(span)Elapsed time since last checkpoint exceeds span
AnyCheckpointPolicy(...)Any sub-policy triggers
+
+

B+Tree

+

B+tree keyed by signed 64-bit long keys. Leaf pages store (key, payload) pairs; interior pages store routing keys and child pointers. Supports forward-only cursor iteration, cache-only fast paths, and page-level rebalance/merge on delete.

+

B1. Create a New B+Tree

+ 
// Allocates a root page and returns its ID
+                uint rootPageId = await BTree.CreateNewAsync(pager);
+                
+                // Open the tree
+                var tree = new BTree(pager, rootPageId);
+
+

B2. Insert a Key-Value Pair

+

Payload is raw bytes -- the B+tree has no opinion on format.

+ 
byte[] payload = System.Text.Encoding.UTF8.GetBytes("Hello, B+tree!");
+                await tree.InsertAsync(key: 42, payload);
+
+

If the leaf page is full, the tree automatically splits the leaf and propagates the split key up to interior pages.

+

B3. Point Lookup

+ 
byte[]? result = await tree.FindAsync(key: 42);
+                if (result is not null)
+                    Console.WriteLine(System.Text.Encoding.UTF8.GetString(result));
+
+

B4. Cache-Only Fast Path

+

Avoids async I/O when all required pages are already cached.

+ 
if (tree.TryFindCached(key: 42, out byte[]? payload))
+                {
+                    // Hit: payload is definitive (null = not found, non-null = value)
+                    Console.WriteLine($"Cache hit: {payload is not null}");
+                }
+                else
+                {
+                    // Miss: need to call FindAsync for full traversal
+                    payload = await tree.FindAsync(key: 42);
+                }
+
+

B5. Delete a Key

+ 
bool deleted = await tree.DeleteAsync(key: 42);
+                Console.WriteLine(deleted ? "Deleted." : "Key not found.");
+
+

B6. Forward Cursor Scan

+

Iterate all entries in key order. The cursor follows leaf-to-leaf next pointers (no interior page I/O after the first leaf).

+ 
var cursor = tree.CreateCursor();
+                while (await cursor.MoveNextAsync())
+                {
+                    long key = cursor.CurrentKey;
+                    ReadOnlyMemory<byte> value = cursor.CurrentValue;
+                    Console.WriteLine($"Key={key}, PayloadSize={value.Length}");
+                }
+
+

B7. Seek to a Specific Key

+

Position the cursor at the first key >= target, then iterate forward.

+ 
var cursor = tree.CreateCursor();
+                if (await cursor.SeekAsync(targetKey: 100))
+                {
+                    do
+                    {
+                        Console.WriteLine($"Key={cursor.CurrentKey}");
+                    } while (await cursor.MoveNextAsync());
+                }
+
+

B8. Count Entries

+

Walks all leaf pages and sums cell counts.

+ 
long count = await tree.CountEntriesAsync();
+                Console.WriteLine($"Tree contains {count} entries.");
+
+
+

Write-Ahead Log (WAL)

+

Redo-style WAL for crash recovery and concurrent snapshot-isolated readers. Each commit writes dirty pages as frames to the WAL file. On checkpoint, committed frames are copied to the main database file.

+ 
WAL File Format:
+                  [WAL Header: 32 bytes]
+                    Magic ─ Version ─ PageSize ─ Checksum salt
+                  [Frame 0: 4120 bytes]
+                    [FrameHeader: 24 bytes] ─ PageId ─ DbPageCount ─ Checksum
+                    [PageData: 4096 bytes]
+                  [Frame 1: 4120 bytes]
+                    ...
+
+

W1. Open or Create a WAL

+ 
var walIndex = new WalIndex();
+                await using var wal = new WriteAheadLog("mydb.cdb", walIndex);
+                await wal.OpenAsync(currentDbPageCount: pager.PageCount);
+                // WAL file: "mydb.cdb.wal"
+
+

W2. Write Transaction to WAL

+ 
wal.BeginTransaction();
+                
+                // Append modified pages as frames
+                await wal.AppendFrameAsync(pageId: 1, pageData);
+                await wal.AppendFrameAsync(pageId: 5, pageData);
+                
+                // Commit: the last frame gets dbPageCount > 0, marking the commit boundary
+                await wal.CommitAsync(newDbPageCount: pager.PageCount);
+                
+                // Or rollback: truncates uncommitted frames
+                await wal.RollbackAsync();
+
+

W3. Take a Reader Snapshot

+

Freeze the WAL index at a point in time for a concurrent reader.

+ 
WalSnapshot snapshot = walIndex.TakeSnapshot();
+                
+                // Reader uses snapshot to resolve page lookups:
+                if (snapshot.TryGet(pageId: 1, out long walOffset))
+                {
+                    byte[] page = await wal.ReadPageAsync(walOffset);
+                    // page is the committed version at snapshot time
+                }
+
+

W4. Checkpoint WAL to Database File

+

Copy all committed WAL pages to the main database file, then reset the WAL.

+ 
await wal.CheckpointAsync(device, pageCount: pager.PageCount);
+                walIndex.Reset();
+                // WAL is now empty; all data is in the main file
+
+
+

Slotted Page Layout

+

SlottedPage is a struct that overlays a byte[4096] buffer, providing structured access to variable-size cells within a fixed-size page.

+ 
[Header: 9 bytes]
+                  PageType (1) ─ CellCount (2) ─ CellContentStart (2) ─ RightChild/NextLeaf (4)
+                [Cell Pointers: 2 bytes each, growing forward]
+                  Offset to cell data for each cell
+                [Free Space]
+                [Cell Data: growing backward from page end]
+                  Variable-size cells packed from the end
+
+

S1. Initialize a Page

+ 
byte[] buffer = new byte[4096];
+                var sp = new SlottedPage(buffer, pageId: 1);
+                sp.Initialize(PageConstants.PageTypeLeaf);
+                
+                Console.WriteLine($"Type: {sp.PageType}");        // Leaf
+                Console.WriteLine($"Cells: {sp.CellCount}");      // 0
+                Console.WriteLine($"Free: {sp.FreeSpace} bytes");  // ~4085
+
+

S2. Insert and Read Cells

+ 
byte[] cellData = new byte[] { 0x01, 0x02, 0x03, 0x04 };
+                bool inserted = sp.InsertCell(index: 0, cellData);
+                Console.WriteLine($"Inserted: {inserted}"); // true
+                
+                Span<byte> cell = sp.GetCell(index: 0);
+                Console.WriteLine($"Cell[0] length: {cell.Length}"); // 4
+
+

S3. Delete a Cell and Defragment

+ 
sp.DeleteCell(index: 0);
+                Console.WriteLine($"Cells after delete: {sp.CellCount}"); // 0
+                
+                // After many inserts/deletes, free space may be fragmented
+                sp.Defragment(); // rewrites cells contiguously at end of page
+
+
+

Indexing

+

Secondary B+tree-backed indexes with optional caching and ordered range scan support.

+ 
public interface IIndexStore
+                {
+                    uint RootPageId { get; }
+                    ValueTask<byte[]?> FindAsync(long key, CancellationToken ct = default);
+                    ValueTask InsertAsync(long key, ReadOnlyMemory<byte> payload, CancellationToken ct = default);
+                    ValueTask<bool> DeleteAsync(long key, CancellationToken ct = default);
+                    IIndexCursor CreateCursor(IndexScanRange range);
+                }
+
+

I1. Create an Index Store

+ 
uint indexRootPage = await BTree.CreateNewAsync(pager);
+                var indexTree = new BTree(pager, indexRootPage);
+                IIndexStore index = new BTreeIndexStore(indexTree);
+
+

I2. Insert and Lookup Index Entries

+

Index payload typically contains the rowid(s) of matching rows.

+ 
// Insert: key = hashed column value, payload = rowid (8 bytes)
+                byte[] rowIdPayload = BitConverter.GetBytes(42L);
+                await index.InsertAsync(key: hashOfColumnValue, rowIdPayload);
+                
+                // Lookup
+                byte[]? result = await index.FindAsync(key: hashOfColumnValue);
+                if (result is not null)
+                {
+                    long rowId = BitConverter.ToInt64(result);
+                    Console.WriteLine($"Found rowid: {rowId}");
+                }
+
+

I3. Range Scan with Cursor

+ 
var range = new IndexScanRange(
+                    LowerBound: 100, LowerInclusive: true,
+                    UpperBound: 200, UpperInclusive: false);
+                
+                var cursor = index.CreateCursor(range);
+                while (await cursor.MoveNextAsync())
+                {
+                    Console.WriteLine($"IndexKey={cursor.CurrentKey}");
+                }
+                
+                // Full scan:
+                var fullCursor = index.CreateCursor(IndexScanRange.All);
+                
+                // Point lookup as cursor:
+                var pointCursor = index.CreateCursor(IndexScanRange.At(42));
+
+

I4. Add Caching to an Index

+

Wrap any IIndexStore with an LRU cache for repeated lookups.

+ 
IIndexStore cached = new CachingIndexStore(
+                    inner: new BTreeIndexStore(indexTree),
+                    capacity: 2048);
+                
+                // Lookups check cache first; inserts/deletes update cache
+                byte[]? result = await cached.FindAsync(key: 42);
+
+
+

Record Serialization

+

Compact binary encoding for database rows. Supports selective column projection and fast filter evaluation without materializing managed strings.

+ 
Binary Format:
+                  [columnCount: varint]
+                  [col0_typeTag: 1 byte] [col0_data: ...]
+                  [col1_typeTag: 1 byte] [col1_data: ...]
+                  ...
+                
+                Type Tags:
+                  Null (0x00)    -> no data
+                  Integer (0x01) -> 8 bytes, little-endian long
+                  Text (0x02)    -> [length: varint] [UTF-8 bytes]
+                  Real (0x03)    -> 8 bytes, little-endian double (IEEE 754)
+                  Blob (0x04)    -> [length: varint] [raw bytes]
+
+

R1. Encode and Decode a Row

+ 
var values = new DbValue[]
+                {
+                    DbValue.FromInteger(1),
+                    DbValue.FromText("Alice"),
+                    DbValue.FromInteger(30)
+                };
+                
+                byte[] encoded = RecordEncoder.Encode(values);
+                DbValue[] decoded = RecordEncoder.Decode(encoded);
+                
+                Console.WriteLine($"Id={decoded[0].AsInteger}, Name={decoded[1].AsText}, Age={decoded[2].AsInteger}");
+
+

R2. Selective Column Projection

+

Decode only the columns you need -- avoids materializing unused fields.

+ 
// Decode only columns 0 and 1 (skip column 2)
+                DbValue[] partial = RecordEncoder.DecodeUpTo(encoded, maxColumnIndexInclusive: 1);
+                
+                // Decode a single column by index
+                DbValue age = RecordEncoder.DecodeColumn(encoded, columnIndex: 2);
+
+

R3. Fast Filter Without Materialization

+

Evaluate filters on encoded rows without allocating managed strings.

+ 
// Check if column 1 equals "Alice" without creating a string
+                byte[] expectedUtf8 = "Alice"u8.ToArray();
+                if (RecordEncoder.TryColumnTextEquals(encoded, columnIndex: 1, expectedUtf8, out bool equals))
+                {
+                    Console.WriteLine($"Column 1 is Alice: {equals}");
+                }
+                
+                // Check numeric column for comparison
+                if (RecordEncoder.TryDecodeNumericColumn(encoded, columnIndex: 2,
+                    out long intValue, out double realValue, out bool isReal))
+                {
+                    Console.WriteLine($"Age: {intValue}");
+                }
+                
+                // Check for null
+                bool isNull = RecordEncoder.IsColumnNull(encoded, columnIndex: 0);
+
+

R4. Varint Encoding

+

Variable-length unsigned integer encoding (LEB128-style). Small values encode in 1 byte; up to 64-bit values supported.

+ 
Span<byte> buffer = stackalloc byte[10];
+                int bytesWritten = Varint.Write(buffer, 300UL);
+                
+                ulong value = Varint.Read(buffer, out int bytesRead);
+                Console.WriteLine($"Value: {value}, Bytes: {bytesRead}"); // 300, 2
+                
+                int predictedSize = Varint.SizeOf(300UL); // 2
+
+
+

Schema Catalog

+

B+tree-backed metadata store for tables, indexes, views, and triggers. Provides in-memory caching with a schema version counter for cache invalidation.

+

C1. Initialize the Catalog

+ 
var catalog = await SchemaCatalog.CreateAsync(pager);
+                Console.WriteLine($"Schema version: {catalog.SchemaVersion}");
+
+

C2. Create and Query Tables

+ 
// Create a table
+                var schema = new TableSchema
+                {
+                    TableName = "users",
+                    Columns = new[]
+                    {
+                        new ColumnDefinition { Name = "id", Type = DbType.Integer, IsPrimaryKey = true },
+                        new ColumnDefinition { Name = "name", Type = DbType.Text },
+                        new ColumnDefinition { Name = "age", Type = DbType.Integer },
+                    }
+                };
+                await catalog.CreateTableAsync(schema);
+                
+                // Query table metadata
+                TableSchema? users = catalog.GetTable("users");
+                uint rootPage = catalog.GetTableRootPage("users");
+                BTree tableTree = catalog.GetTableTree("users");
+                
+                // List all tables
+                IReadOnlyCollection<string> tableNames = catalog.GetTableNames();
+                
+                // For snapshot readers
+                BTree snapshotTree = catalog.GetTableTree("users", snapshotPager);
+
+

C3. Create and Query Indexes

+ 
var indexSchema = new IndexSchema
+                {
+                    IndexName = "idx_users_name",
+                    TableName = "users",
+                    Columns = new[] { "name" },
+                    IsUnique = false,
+                };
+                await catalog.CreateIndexAsync(indexSchema);
+                
+                // Get index store
+                IIndexStore indexStore = catalog.GetIndexStore("idx_users_name");
+                
+                // List indexes for a table
+                IReadOnlyList<IndexSchema> indexes = catalog.GetIndexesForTable("users");
+                
+                // For snapshot readers
+                IIndexStore snapshotIndex = catalog.GetIndexStore("idx_users_name", snapshotPager);
+
+

C4. Views and Triggers

+ 
// Views
+                await catalog.CreateViewAsync("active_users", "SELECT * FROM users WHERE age > 18");
+                string? viewSql = catalog.GetViewSql("active_users");
+                bool isView = catalog.IsView("active_users");
+                
+                // Triggers
+                var trigger = new TriggerSchema
+                {
+                    TriggerName = "trg_users_audit",
+                    TableName = "users",
+                    Event = TriggerEvent.AfterInsert,
+                    Body = "INSERT INTO audit_log (table_name, action) VALUES ('users', 'INSERT')",
+                };
+                await catalog.CreateTriggerAsync(trigger);
+                
+                IReadOnlyList<TriggerSchema> triggers = catalog.GetTriggersForTable("users");
+
+

C5. Persist Root Page Changes

+

After B+tree operations that split the root page, persist the new root page ID in the catalog.

+ 
// Persist for a single table + its indexes (fast)
+                await catalog.PersistRootPageChangesAsync("users");
+                
+                // Persist for all tables and indexes (slower, used during batch operations)
+                await catalog.PersistAllRootPageChangesAsync();
+
+
+

Folder & File Storage

+

FileStorageDevice is a raw byte device. To build a folder/file storage system on top of it, use the higher-level Database + Collection<T> API from CSharpDB.Engine. A single .cdb file (backed by one FileStorageDevice) holds all folders and files as typed collection documents in B+tree-backed collections.

+
+

Add the Engine reference to your project if it is not already there:

+ 
<ProjectReference Include="..\CSharpDB.Engine\CSharpDB.Engine.csproj" />
+
+
+
+

Domain Models

+

Define records that represent a folder and a file entry. These are serialized as JSON by Collection<T>.

+ 
public record FolderEntry(
+                    string Name,
+                    string Path,              // e.g. "/documents/reports"
+                    DateTime CreatedAt,
+                    string? Description = null);
+                
+                public record FileEntry(
+                    string Name,
+                    string FolderPath,        // parent folder path
+                    string Content,           // UTF-8 text content (or Base64 for binary)
+                    string ContentType,       // e.g. "text/plain", "application/json"
+                    long SizeBytes,
+                    DateTime CreatedAt,
+                    DateTime UpdatedAt);
+
+

Keys follow a path convention:

+
    +
  • Folders -> "/documents/reports"
    • +
  • Files -> "/documents/reports/summary.txt"
    • +
+

F1. Bootstrap the Storage

+

Open (or create) a single .cdb file and obtain the two collections.

+ 
await using var db = await Database.OpenAsync("storage.cdb");
+                var folders = await db.GetCollectionAsync<FolderEntry>("folders");
+                var files   = await db.GetCollectionAsync<FileEntry>("files");
+
+

Everything -- the B+tree pages, the WAL, and the page cache -- is managed by the single FileStorageDevice that Database.OpenAsync creates internally.

+

For a long-lived process that should keep touched pages hot while remaining durable on disk, use hybrid lazy-resident mode instead:

+ 
await using var db = await Database.OpenHybridAsync(
+                    "storage.cdb",
+                    new DatabaseOptions(),
+                    new HybridDatabaseOptions
+                    {
+                        PersistenceMode = HybridPersistenceMode.IncrementalDurable,
+                        HotCollectionNames = ["folders", "files"]
+                    });
+
+

That opens from disk lazily, keeps touched pages resident by cache policy, and can preload selected hot collections into the hybrid pager cache at startup.

+

F2. Create a Folder

+

Use the folder's path as the collection key so point lookups stay efficient; collection keys are hashed internally before probing the backing trees.

+ 
async Task CreateFolderAsync(string path, string? description = null)
+                {
+                    var entry = new FolderEntry(
+                        Name:        Path.GetFileName(path.TrimEnd('/')),
+                        Path:        path,
+                        CreatedAt:   DateTime.UtcNow,
+                        Description: description);
+                
+                    await folders.PutAsync(path, entry);
+                }
+                
+                await CreateFolderAsync("/documents");
+                await CreateFolderAsync("/documents/reports", description: "Monthly reports");
+                await CreateFolderAsync("/images");
+
+

F3. Create a File Inside a Folder

+

The key is the full file path, which guarantees uniqueness across all folders.

+ 
async Task CreateFileAsync(string folderPath, string fileName, string content, string contentType = "text/plain")
+                {
+                    string key = $"{folderPath}/{fileName}";
+                    var entry = new FileEntry(
+                        Name:        fileName,
+                        FolderPath:  folderPath,
+                        Content:     content,
+                        ContentType: contentType,
+                        SizeBytes:   System.Text.Encoding.UTF8.GetByteCount(content),
+                        CreatedAt:   DateTime.UtcNow,
+                        UpdatedAt:   DateTime.UtcNow);
+                
+                    await files.PutAsync(key, entry);
+                }
+                
+                await CreateFileAsync("/documents/reports", "q1.txt",  "Q1 earnings: $1.2M");
+                await CreateFileAsync("/documents/reports", "q2.txt",  "Q2 earnings: $1.5M");
+                await CreateFileAsync("/documents",          "notes.md", "# Notes\nTodo list...", "text/markdown");
+                await CreateFileAsync("/images",             "logo.svg", "<svg>...</svg>",        "image/svg+xml");
+
+

F4. Read a File

+

Retrieve a file by its full path key.

+ 
FileEntry? file = await files.GetAsync("/documents/reports/q1.txt");
+                
+                if (file is not null)
+                {
+                    Console.WriteLine($"Name:    {file.Name}");
+                    Console.WriteLine($"Type:    {file.ContentType}");
+                    Console.WriteLine($"Size:    {file.SizeBytes} bytes");
+                    Console.WriteLine($"Content: {file.Content}");
+                }
+                else
+                {
+                    Console.WriteLine("File not found.");
+                }
+
+

F5. List All Files in a Folder

+

Collection<T>.FindAsync performs a full scan with an in-memory predicate -- suitable for small-to-medium collections.

+ 
string targetFolder = "/documents/reports";
+                await foreach (var kvp in files.FindAsync(f => f.FolderPath == targetFolder))
+                {
+                    Console.WriteLine($"  {kvp.Value.Name}  ({kvp.Value.SizeBytes} bytes)  [{kvp.Value.UpdatedAt:u}]");
+                }
+
+

F6. List All Folders

+ 
await foreach (var kvp in folders.ScanAsync())
+                {
+                    var f = kvp.Value;
+                    Console.WriteLine($"{f.Path,-40} created {f.CreatedAt:u}");
+                }
+
+

F7. Update File Content

+

PutAsync is an upsert -- it replaces the document at the key if it already exists.

+ 
async Task UpdateFileAsync(string filePath, string newContent)
+                {
+                    FileEntry? existing = await files.GetAsync(filePath);
+                    if (existing is null) throw new FileNotFoundException($"File not found: {filePath}");
+                
+                    var updated = existing with
+                    {
+                        Content   = newContent,
+                        SizeBytes = System.Text.Encoding.UTF8.GetByteCount(newContent),
+                        UpdatedAt = DateTime.UtcNow
+                    };
+                
+                    await files.PutAsync(filePath, updated);
+                }
+                
+                await UpdateFileAsync("/documents/reports/q1.txt", "Q1 earnings: $1.4M (revised)");
+
+

F8. Delete a File

+ 
bool deleted = await files.DeleteAsync("/documents/reports/q2.txt");
+                Console.WriteLine(deleted ? "File deleted." : "File not found.");
+
+

F9. Delete a Folder and Its Contents

+

There is no cascading delete built in, so collect the child keys first, then delete in a single transaction.

+ 
async Task DeleteFolderAsync(string folderPath)
+                {
+                    // Collect all file keys under this folder
+                    var toDelete = new List<string>();
+                    await foreach (var kvp in files.FindAsync(f => f.FolderPath.StartsWith(folderPath, StringComparison.Ordinal)))
+                        toDelete.Add(kvp.Key);
+                
+                    await db.BeginTransactionAsync();
+                    try
+                    {
+                        foreach (var key in toDelete)
+                            await files.DeleteAsync(key);
+                        await folders.DeleteAsync(folderPath);
+                        await db.CommitAsync();
+                    }
+                    catch
+                    {
+                        await db.RollbackAsync();
+                        throw;
+                    }
+                }
+                
+                await DeleteFolderAsync("/documents/reports");
+
+

F10. Rename or Move a File

+

CSharpDB does not have a rename primitive; copy the document to the new key and delete the old one inside a transaction.

+ 
async Task MoveFileAsync(string sourcePath, string destinationPath)
+                {
+                    FileEntry? source = await files.GetAsync(sourcePath);
+                    if (source is null) throw new FileNotFoundException($"Source not found: {sourcePath}");
+                
+                    string newFolder   = Path.GetDirectoryName(destinationPath)!.Replace('\\', '/');
+                    string newFileName = Path.GetFileName(destinationPath);
+                    var moved = source with
+                    {
+                        Name       = newFileName,
+                        FolderPath = newFolder,
+                        UpdatedAt  = DateTime.UtcNow
+                    };
+                
+                    await db.BeginTransactionAsync();
+                    try
+                    {
+                        await files.PutAsync(destinationPath, moved);
+                        await files.DeleteAsync(sourcePath);
+                        await db.CommitAsync();
+                    }
+                    catch
+                    {
+                        await db.RollbackAsync();
+                        throw;
+                    }
+                }
+                
+                await MoveFileAsync("/documents/notes.md", "/documents/reports/notes.md");
+
+

F11. Search Files by Predicate

+

Find all Markdown files larger than 100 bytes modified after a given date.

+ 
DateTime since = new DateTime(2025, 1, 1, 0, 0, 0, DateTimeKind.Utc);
+                await foreach (var kvp in files.FindAsync(f =>
+                    f.ContentType == "text/markdown" &&
+                    f.SizeBytes   > 100             &&
+                    f.UpdatedAt   > since))
+                {
+                    Console.WriteLine($"{kvp.Key}  ({kvp.Value.SizeBytes} bytes)");
+                }
+
+

F12. Bulk Create with an Explicit Transaction

+

Wrap multiple writes in a single transaction so they all succeed or all roll back together.

+ 
string[] reportNames = ["jan.txt", "feb.txt", "mar.txt", "apr.txt"];
+                
+                await db.BeginTransactionAsync();
+                try
+                {
+                    await CreateFolderAsync("/archive/2025");
+                    foreach (var name in reportNames)
+                        await CreateFileAsync("/archive/2025", name, $"Report: {name}");
+                    await db.CommitAsync();
+                    Console.WriteLine($"Committed {reportNames.Length} files in one transaction.");
+                }
+                catch
+                {
+                    await db.RollbackAsync();
+                    throw;
+                }
+
+

F13. SQL-Based Approach

+

If you prefer a relational model, create folders and files tables with SQL and use ExecuteAsync.

+ 
await using var db = await Database.OpenAsync("storage.cdb");
+                
+                // Create schema
+                await db.ExecuteAsync("""
+                    CREATE TABLE IF NOT EXISTS folders (
+                        id          INTEGER PRIMARY KEY,
+                        path        TEXT NOT NULL,
+                        name        TEXT NOT NULL,
+                        description TEXT,
+                        created_at  TEXT NOT NULL
+                    )
+                    """);
+                
+                await db.ExecuteAsync("""
+                    CREATE TABLE IF NOT EXISTS files (
+                        id           INTEGER PRIMARY KEY,
+                        folder_path  TEXT NOT NULL,
+                        name         TEXT NOT NULL,
+                        content      TEXT NOT NULL,
+                        content_type TEXT NOT NULL,
+                        size_bytes   INTEGER NOT NULL,
+                        created_at   TEXT NOT NULL,
+                        updated_at   TEXT NOT NULL
+                    )
+                    """);
+                
+                // Insert a folder
+                await db.ExecuteAsync("""
+                    INSERT INTO folders (path, name, created_at)
+                    VALUES ('/documents', 'documents', '2025-01-01T00:00:00Z')
+                    """);
+                
+                // Insert a file
+                await db.ExecuteAsync("""
+                    INSERT INTO files (folder_path, name, content, content_type, size_bytes, created_at, updated_at)
+                    VALUES ('/documents', 'readme.txt', 'Hello world', 'text/plain', 11,
+                            '2025-01-01T00:00:00Z', '2025-01-01T00:00:00Z')
+                    """);
+                
+                // Query files in a folder
+                var result = await db.ExecuteAsync("SELECT name, size_bytes FROM files WHERE folder_path = '/documents'");
+                foreach (var row in result.Rows)
+                    Console.WriteLine($"{row[0]}  ({row[1]} bytes)");
+
+

F14. One Database File per Folder (Multi-Volume)

+

Map each top-level folder to its own .cdb file. Each file gets its own FileStorageDevice instance, giving you independent WAL, checkpoint, and locking per folder.

+ 
// Each folder is a separate database file
+                var volumes = new Dictionary<string, Database>(StringComparer.Ordinal);
+                
+                async ValueTask<Database> GetVolumeAsync(string folderName)
+                {
+                    if (!volumes.TryGetValue(folderName, out var db))
+                    {
+                        db = await Database.OpenAsync($"{folderName}.cdb");
+                        volumes[folderName] = db;
+                    }
+                    return db;
+                }
+                
+                // Write to the "documents" volume
+                var docsDb   = await GetVolumeAsync("documents");
+                var docsFiles = await docsDb.GetCollectionAsync<FileEntry>("files");
+                await docsFiles.PutAsync("readme.txt", new FileEntry(
+                    Name:        "readme.txt",
+                    FolderPath:  "/",
+                    Content:     "Welcome to the documents volume.",
+                    ContentType: "text/plain",
+                    SizeBytes:   32,
+                    CreatedAt:   DateTime.UtcNow,
+                    UpdatedAt:   DateTime.UtcNow));
+                
+                // Write to the "images" volume
+                var imagesDb    = await GetVolumeAsync("images");
+                var imagesFiles = await imagesDb.GetCollectionAsync<FileEntry>("files");
+                await imagesFiles.PutAsync("logo.svg", new FileEntry(
+                    Name:        "logo.svg",
+                    FolderPath:  "/",
+                    Content:     "<svg>...</svg>",
+                    ContentType: "image/svg+xml",
+                    SizeBytes:   14,
+                    CreatedAt:   DateTime.UtcNow,
+                    UpdatedAt:   DateTime.UtcNow));
+                
+                // Dispose all volumes on shutdown
+                foreach (var (_, volume) in volumes)
+                    await volume.DisposeAsync();
+
+
+

When to use multi-volume: large datasets where you want per-folder backup, different checkpoint intervals, or parallel writes to disjoint folders. For most use-cases a single .cdb file is simpler.

+
+
+

Key Design Notes

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ConcernDetail
No shared file pointerRandomAccess APIs are stateless w.r.t. position, so concurrent reads at different offsets are safe without locking.
Async-firstAll I/O is issued via RandomAccess.ReadAsync / WriteAsync, keeping the storage graph on the platform async file-I/O path.
Zero-fill on short readsReadAsync always fills the entire buffer. Pages beyond EOF are returned as zeros, matching an uninitialized page convention used by the Pager.
fsync on flushFlushAsync calls RandomAccess.FlushToDisk which maps to FlushFileBuffers (Windows) or fsync (Linux/macOS), guaranteeing crash durability.
FileShare.ReadOther processes can open the file read-only concurrently; write access is exclusive to the owning FileStorageDevice instance.
IDisposable + IAsyncDisposableBoth patterns are supported; prefer await using in async code.
4 KB page sizeAll pages are PageConstants.PageSize (4096 bytes). Page 0 reserves 100 bytes for the file header.
Single writer, multiple readersThe TransactionCoordinator enforces a single writer via SemaphoreSlim. Readers use WAL snapshots for isolation.
Optional memory-mapped readsThe pager can use memory-mapped reads for clean main-file pages when PagerOptions.UseMemoryMappedReads is enabled and the storage device supports it.
Sequential scan read-aheadThe pager can speculatively pull the next B+tree leaf page during forward scans when EnableSequentialLeafReadAhead is enabled.
Checkpoint residency preservationWith PagerOptions.PreserveOwnedPagesOnCheckpoint, already-owned main-file pages can stay resident across checkpoint. This is what the engine's lazy-resident hybrid mode relies on.
B+tree leaf linkingLeaf pages are linked via RightChildOrNextLeaf pointers, enabling efficient forward-only cursor scans without interior I/O.
Pluggable checkpoint policiesICheckpointPolicy allows frame-count, WAL-size, time-interval, or custom composite triggers.
Schema versioningSchemaCatalog.SchemaVersion increments on every DDL operation, enabling cache invalidation in upper layers.
Interceptor pipelineIPageOperationInterceptor provides hooks for diagnostics, metrics, and custom behavior on page reads, writes, transactions, and checkpoints.
+
+

See Also

+
    +
  • Architecture Guide
    • +
  • Storage Package README
    • +
  • Storage Tutorial Track
    • +
  • Advanced Storage Examples
    • +
  • Engine README
    • +
  • Storage Diagnostics README
    • +
  • Benchmark Suite
    • +
  • Roadmap
    • +
+
+
+
+ + + + + diff --git a/www/docs/storage-engine.html b/www/docs/storage-engine.html index 616c0fd9..e756a300 100644 --- a/www/docs/storage-engine.html +++ b/www/docs/storage-engine.html @@ -64,6 +64,9 @@

On This Page

CSharpDB.Storage

A low-level, high-performance storage engine for .NET 10 built on top of RandomAccess and SafeFileHandle. It provides random-access async I/O, page caching, write-ahead logging (WAL), crash recovery, and the B+tree/index primitives that power the SQL engine and collection API.

+
+ Need the full source guide? The original long-form markdown version is preserved as Storage Engine Source Reference. +

Architecture Overview

diff --git a/www/roadmap-reference.html b/www/roadmap-reference.html new file mode 100644 index 00000000..eeb5deb6 --- /dev/null +++ b/www/roadmap-reference.html @@ -0,0 +1,558 @@ + + + + + + + + Roadmap Source Reference — CSharpDB + + + + + + + + + + + + + + +
+
+ + +
+
Source reference. This page preserves the original long-form markdown content that previously lived at docs/roadmap.md. For the shorter curated page, see Roadmap.
+

CSharpDB Roadmap

+

This document outlines the planned direction for CSharpDB, organized by timeframe and priority. Items are roughly ordered by expected impact within each tier, and statuses are intended to reflect the current v3.4.0 state of the repo.

+
+

Near-Term

+

Recently completed improvements to query performance, storage/runtime behavior, maintenance workflows, and developer ergonomics.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FeatureDescriptionStatus
DISTINCT keywordDeduplicate rows in SELECT outputDone
Composite indexesMulti-column indexes for covering more query patternsDone
Index range scansUse indexes for <, >, <=, >=, BETWEEN — not just equalityDone
Prepared statement cacheCache parsed ASTs and query plans to avoid re-parsing identical SQLDone
Cached max rowidAvoid repeated O(n) scans when generating row IDs on insert (in-memory + persisted high-water mark)Done
B+tree delete rebalancingMerge underflowed pages on delete to reclaim spaceDone
In-memory database modeOpen a database fully in memory, load a disk database into memory, and save a committed snapshot back to diskDone
Shared in-memory ADO.NET modeSupport Data Source=:memory: and named shared in-memory databases with explicit save/loadDone
Collection field indexesEquality-based secondary indexes for Collection<T> via EnsureIndexAsync / FindByIndexAsyncDone
Reader session reuseReuse snapshot pager and query planner inside ReaderSession for burst concurrent readsDone
Architecture enforcementCSharpDB.Client is now the main caller-facing interaction layer across local and remote scenarios; ADO.NET now routes ordinary direct and daemon-backed access through that layer, with only named shared in-memory provider state still retaining an internal engine dependencyDone
Database administrationMaintenance report, reindex (database/table/index/collection), VACUUM/compact, fragmentation analysis, database size reportDone
Dedicated gRPC daemonCSharpDB.Daemon host plus CSharpDB.Client gRPC coverage for SQL, schema, procedures, collections, and maintenanceDone
Storage tuning presetsUseLookupOptimizedPreset() and UseWriteOptimizedPreset() for file-backed workloadsDone
Memory-mapped main-file readsOpt-in mapped clean-page reads plus copy-on-write materialization for mutable access on file-backed databasesDone
Background WAL checkpointingIncremental/sliced auto-checkpointing to move work off the triggering commitDone
SQL executor/read-path fast pathsCompact scan and indexed-range projections, broader join lookup/covered paths, grouped/composite index aggregates, correlated subquery filter fast paths, and lower row materialization overheadDone
Table/index statisticsANALYZE command with persisted row counts, column NDV/min/max, stale tracking, and initial stats-guided index selection in the query plannerDone
Collection binary payloadsBinary direct-payload format with faster hydration, direct field/path extraction, and richer path-based indexingDone
Collection path indexesNested scalar, array-element, nested array-object, Guid, temporal, and ordered text path indexes with FindByPathAsync / FindByPathRangeAsyncDone
Hybrid storage modeLazy-resident durable storage with gRPC tunable file-cache configuration; Admin direct local hosting keeps a warm in-process database instance and uses hybrid incremental-durable options by defaultDone
Client backup/restoreBackupAsync / RestoreAsync as first-class ICSharpDbClient operations across direct, HTTP, gRPC, CLI, and AdminDone
Older DB foreign-key retrofit migrationValidate/apply maintenance workflow that rewrites existing child tables with persisted FK metadata across direct, HTTP, gRPC, CLI, and AdminDone
+
+

Mid-Term

+

SQL feature parity, provider/tooling compatibility, and ecosystem expansion.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FeatureDescriptionStatus
User-defined functionsBroader built-in scalar function registry (UPPER, ABS, COALESCE, etc.), user-registered C# functions, native plugin extensionsPlanned
SubqueriesScalar subqueries, IN (SELECT ...), EXISTS (SELECT ...), including correlated evaluation in WHERE, non-aggregate projection, and UPDATE/DELETE expressionsDone
UNION / INTERSECT / EXCEPTSet operations across SELECT results, including use in top-level queries, views, and CTE query bodiesDone
Window functionsROW_NUMBER(), RANK(), DENSE_RANK(), LEAD(), LAG()Planned
DEFAULT column valuesAllow default expressions in column definitionsPlanned
CHECK constraintsArbitrary expression-based constraints per column or per tablePlanned
Foreign key constraintsv1 support for single-column, column-level REFERENCES with optional ON DELETE CASCADE, plus sys.foreign_keys and metadata/tooling surfacesDone
Remote host consolidationCSharpDB.Daemon now hosts the existing REST/HTTP /api surface and gRPC from one long-running process backed by the same warm daemon-hosted client; standalone CSharpDB.Api remains supported for REST-only hostingDone
Remote host securityAdd built-in authentication, authorization, and transport-security options for remote HTTP and gRPC access, including API keys, protected admin endpoints, and TLS/mTLS deployment supportPlanned
Daemon service packagingPackage the existing CSharpDB.Daemon host as a persistent background service across systemd, Windows Service, and launchdDone
Cross-platform deploymentSelf-contained daemon archives and install scripts ship for Windows, Linux, and macOS; dotnet tool, Docker, Homebrew, and winget distribution remain future workIn Progress
NuGet packagePublish and maintain CSharpDB.Engine, CSharpDB.Data, CSharpDB.Client, and CSharpDB.Primitives as the primary NuGet packagesDone
Connection poolingPool underlying direct embedded sessions behind CSharpDbConnection to amortize open/close costDone
Admin dashboard improvementsRicher SQL editor UX, query history, deeper diagnostics, and integrated Forms/Reports tooling beyond the core schema/procedure/storage surfaceDone
Visual query designerClassic Admin query builder with source canvas, join editing, design grid, SQL preview, and saved designer layoutsDone
ETL pipelinesBuilt-in package-driven pipeline runtime with validation, dry-run, execute/resume flows, API/CLI/client coverage, run history, and Admin visual designer supportDone
VS Code extensionSchema explorer, SQL editor with IntelliSense, data browser, table designer, storage diagnosticsDone
ADO.NET GetSchema collectionsImplement DbConnection.GetSchema() for standard metadata collections (MetaDataCollections, Tables, Columns, Indexes, Views, ForeignKeys) to support ORMs and tooling that discover schema through ADO.NETDone
Multilingual text supportBINARY, NOCASE, NOCASE_AI, and built-in ICU:<locale> collation now work across SQL schema/query semantics, metadata surfaces, and collection path indexes; dedicated ordered SQL text index optimization remains plannedDone
+
+

Long-Term

+

Advanced features and fundamental architecture enhancements.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FeatureDescriptionStatus
Full-text searchInverted index with tokenization, stemming, and relevance rankingDone
JSON path queryingQuery into JSON document fields in the Collection API (e.g., $.address.city) via FindByPathAsync / FindByPathRangeAsyncDone
Advanced collection storage pathBinary direct-payload format with direct binary hydration, path-based field extraction, and richer expression/path indexesDone
SQL batched row transportInternal row-batch transport serves as the batch-first SQL execution foundation across batch-capable result boundaries, scans, joins, and generic aggregatesDone
Source-generated collection fast pathIn progress: GetGeneratedCollectionAsync<T>(...), generated field descriptors/index bindings, analyzer-packaged collection model/codecs, trim/NativeAOT smoke coverage, and a dedicated sample are now in place while broader package ergonomics and remaining generator coverage continueIn Progress
Page-level compressionCompress cell content within pages to reduce I/O and storagePlanned
At-rest encryptionEncrypt database and WAL files with passphrase-based key management and explicit plaintext/encrypted migration/export pathsResearch
Advanced cost-based query optimizerIn progress: phase-2 stats-guided costing is now in place through internal equi-depth histograms, heavy hitters, composite-index prefix distinct-count summaries, skew-aware lookup/filter estimates, correlation-aware composite equality filters/joins, and bounded DP reordering for small inner-join chains; adaptive re-optimization and public histogram inspection remain future workIn Progress
Async I/O batchingIn progress: WAL frame-chunk writes, chunked checkpoint page copies, shared snapshot/export batching, and reusable B-tree copy utilities now cover the main storage and maintenance write paths; remaining auditing is outside the WAL hot pathIn Progress
Low-latency durable writesDone in v2.9.0: advisory planner-stat persistence can stay deferred without weakening committed-row durability, and sys.table_stats.row_count_is_exact now makes exact versus estimated row-count semantics explicit to planner and COUNT(*) fast pathsDone
Group commit / deferred WAL flushDone in v2.9.0: opt-in UseDurableCommitBatchWindow(...) batches durable WAL flushes across contending in-process transactions and remains an expert measure-first knob rather than default behaviorDone
Initial multi-writer supportExplicit WriteTransaction conflict-detected retry flow, shared auto-commit non-insert isolation, and opt-in ConcurrentWriteTransactions for shared implicit insertsDone
Broader multi-writer insert optimizationImprove hot insert fan-in, row-id reservation, and other high-contention patterns beyond the current initial multi-writer pathResearch
Replication / change feedStream committed changes for read replicas or event-driven architecturesResearch
WebAssembly sandboxed UDFsExecute untrusted user-submitted functions in a WASM sandbox with resource limits (fuel, memory caps) via WasmtimeResearch
+
+

Current Limitations

+

These are known simplifications in the current implementation:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AreaLimitation
FunctionsVery limited scalar function surface today: built-in TEXT(expr) plus aggregate functions; no broader built-in function library or user-defined functions yet
QueryScalar/IN/EXISTS subqueries are supported, including correlated cases in WHERE, non-aggregate projection, and UPDATE/DELETE expressions; correlated subqueries are not yet supported in JOIN ON, GROUP BY, HAVING, ORDER BY, or aggregate projections
QueryUNION, INTERSECT, and EXCEPT are supported; UNION ALL is not implemented yet
QueryNo window functions
SchemaNo SQL DEFAULT column values or CHECK constraints yet. Foreign keys are currently v1 only: single-column, column-level REFERENCES with optional ON DELETE CASCADE; table-level/composite/deferred foreign keys and ON UPDATE actions are not implemented
IndexesEquality lookups support current INTEGER/TEXT indexes, but ordered range-scan pushdown is still limited to single-column INTEGER index paths
RowIdLegacy table schemas without persisted high-water metadata may pay a one-time key scan on first insert
CollectionsFindByIndexAsync supports declared field-equality lookups; FindByPathAsync and FindByPathRangeAsync support path-based queries on indexed paths; FindAsync remains a full scan for unindexed predicates
NetworkingCSharpDB.Daemon now hosts both REST and gRPC from one process; named pipes remain reserved but are not implemented end to end today
SecurityRemote HTTP and gRPC deployment still rely on external network controls or front-end TLS termination; built-in authentication, authorization, and TLS/mTLS support are still planned
Text / MultilingualText is stored as UTF-8 and supports all Unicode languages; default semantics remain ordinal, but opt-in BINARY, NOCASE, NOCASE_AI, and ICU:<locale> collation are implemented for SQL and collection indexes. Dedicated ordered SQL text index optimization remains planned
ConcurrencyThe physical WAL commit path is still serialized at the storage boundary. Initial multi-writer support is shipped, but observed gains still depend on conflict shape and whether shared auto-commit INSERT is left on the default serialized path
StorageNo page-level compression
StorageNo at-rest encryption for database/WAL files; on-disk storage is plaintext only
StorageMemory-mapped reads are opt-in and currently apply only to clean main-file pages; WAL-backed reads still rely on the WAL/cache path
StorageBy default, durable auto-commit single-row writes still pay a physical WAL flush per commit; opt-in UseDurableCommitBatchWindow(...) can trade some commit latency for higher throughput across contending in-process writers, but default behavior remains per-commit durable
QueryPhase-2 cost-based planning is largely in place: ANALYZE, sys.table_stats, sys.column_stats, internal histograms/heavy hitters/prefix stats, and bounded small-chain join reordering now feed join/access-path costing; remaining work is adaptive re-optimization and public histogram/diagnostic surfacing rather than missing core stats-guided costing
QueryInternal row-batch transport is now the default scan-heavy execution foundation across batch-capable scans, joins, aggregates, and result boundaries; remaining work is broader kernel specialization and optional SIMD-style tuning rather than missing core batch coverage
+
+

Completed Milestones

+

Major features already implemented:

+
    +
  • Single-file database with 4 KB page-oriented storage
    • +
  • B+tree-backed tables and secondary indexes
    • +
  • Write-Ahead Log with crash recovery and auto-checkpoint
    • +
  • Concurrent snapshot-isolated readers via WAL-based MVCC
    • +
  • Full SQL pipeline: tokenizer, parser, query planner, operator tree
    • +
  • JOINs (INNER, LEFT, RIGHT, CROSS), aggregates, GROUP BY, HAVING, CTEs
    • +
  • Set operations: UNION, INTERSECT, EXCEPT
    • +
  • SELECT DISTINCT and DISTINCT aggregates
    • +
  • Scalar subqueries, IN (SELECT ...), and EXISTS (SELECT ...), including correlated evaluation in filters, non-aggregate projections, and UPDATE/DELETE expressions
    • +
  • Scalar TEXT(expr) for filter-friendly text coercion
    • +
  • Composite (multi-column) indexes
    • +
  • Ordered integer index range scans (<, <=, >, >=, BETWEEN) in the fast lookup path
    • +
  • ANALYZE, persisted sys.table_stats / sys.column_stats, and stale-aware column-stat refresh
    • +
  • Phase-2 cost-based query planning: statistics-guided access-path selection, join method choice, hash build-side choice, histogram/heavy-hitter/cardinality estimation, composite-prefix correlation modeling, and bounded small-chain inner-join reordering
    • +
  • SQL statement and SELECT plan caching
    • +
  • First-class IDENTITY / AUTOINCREMENT support for INTEGER PRIMARY KEY columns
    • +
  • Persisted table NextRowId high-water mark with compatibility fallback for legacy metadata
    • +
  • Views and triggers (BEFORE/AFTER on INSERT/UPDATE/DELETE)
    • +
  • Foreign key constraints: single-column, column-level REFERENCES with optional ON DELETE CASCADE
    • +
  • Older-database foreign-key retrofit migration across direct, HTTP, gRPC, CLI, and Admin
    • +
  • ADO.NET provider (DbConnection, DbCommand, DbDataReader, DbTransaction)
    • +
  • ADO.NET GetSchema() metadata collections for MetaDataCollections, Tables, Columns, Indexes, Views, and ForeignKeys
    • +
  • ADO.NET connection pooling with ClearPool / ClearAllPools
    • +
  • In-memory database mode with explicit load-from-disk and save-to-disk APIs
    • +
  • Shared/private in-memory ADO.NET connections with named shared-memory hosts
    • +
  • Document Collection API (NoSQL) with typed Put/Get/Delete/Scan/Find
    • +
  • Collection UTF-8 payload fast path with compatibility for legacy backing rows
    • +
  • Collection secondary field indexes via EnsureIndexAsync / FindByIndexAsync
    • +
  • Maintenance report, REINDEX, and VACUUM flows across client, CLI, API, and Admin UI
    • +
  • Dedicated CSharpDB.Daemon gRPC host for remote CSharpDB.Client access
    • +
  • Remote host consolidation in CSharpDB.Daemon, with REST /api and gRPC sharing the same warm hosted database client
    • +
  • Storage tuning presets, bounded WAL read caching, memory-mapped main-file reads, and sliced background WAL auto-checkpointing
    • +
  • SQL executor/read-path fast paths for compact projections, broader join/index coverage, grouped aggregates, and correlated subquery filters
    • +
  • Batch-first SQL row-batch execution foundation with batch-aware scan/index/join roots, shared predicate/projection kernels, and batch-native generic aggregate paths
    • +
  • Interactive CLI with meta-commands and file execution
    • +
  • REST API with 34 endpoints and OpenAPI/Scalar documentation
    • +
  • Blazor Server admin dashboard
    • +
  • Integrated Admin Forms and Reports designers with runtime preview/entry, database-backed metadata persistence, and print-ready report output
    • +
  • B+tree delete rebalancing with underflow handling (borrow/merge + interior collapse path)
    • +
  • Reusable snapshot reader sessions for higher concurrent-read throughput
    • +
  • Comprehensive benchmark suite (micro, macro, stress, scaling, in-memory, shared-memory)
    • +
  • Binary direct-payload collection storage with direct hydration and field/path extraction
    • +
  • Collection path indexes: nested scalar, array-element, nested array-object, Guid, temporal, ordered text
    • +
  • Collection path query APIs: FindByPathAsync and FindByPathRangeAsync
    • +
  • Source-generated typed collection fast path foundations: generated collection models/codecs/field descriptors, trim-safe GetGeneratedCollectionAsync<T>(...), generator diagnostics, NativeAOT trim-smoke validation, and a dedicated sample
    • +
  • Full-text search with tokenization, stemming, and relevance ranking
    • +
  • Hybrid storage mode with lazy-resident durable storage and gRPC tunable file-cache
    • +
  • Client-wide BackupAsync / RestoreAsync across direct, HTTP, gRPC, CLI, and Admin
    • +
  • ReplaceAsync for index stores
    • +
  • Package-driven ETL pipelines with validation, dry-run, execute/resume, persisted run history, and Admin visual designer support
    • +
+
+

See Also

+
    +
  • Architecture Guide — How the engine is structured
    • +
  • Internals & Contributing — How to extend the engine
    • +
  • Deployment & Installation Plan — Cross-platform distribution via dotnet tool, Docker, Homebrew, winget, and install scripts
    • +
  • Multi-Writer Follow-Up Plan — Post-initial multi-writer roadmap, insert-path gaps, and release criteria for broader completion
    • +
  • Query And Durable Write Performance Plan — Combined optimizer phase-2 plus durable-write completion plan, shipped state, and remaining benchmark/future-work boundaries
    • +
  • Multilingual Text Support Plan — Build on existing Unicode text storage with case-insensitive matching, locale-aware sorting, and COLLATE clause support for queries and index definitions
    • +
  • Database Encryption Plan — Encrypted storage format, key management, migration, and managed-surface rollout
    • +
  • Storage Engine Guide — CSharpDB.Storage API reference: device, pager, B+tree, WAL, indexing, serialization, and catalog
    • +
  • Native FFI Tutorials — Python and Node.js examples using the NativeAOT shared library
    • +
  • User-Defined Functions Plan — C# library functions callable by the database, native plugin extensions, and WASM sandboxing
    • +
  • Pub/Sub Change Events Plan — Engine-level change events with channel-based delivery for real-time data subscriptions
    • +
  • Benchmark Suite — Performance data informing optimization priorities
    • +
+
+
+
+ + + + + diff --git a/www/roadmap.html b/www/roadmap.html index d1d74500..90f4863a 100644 --- a/www/roadmap.html +++ b/www/roadmap.html @@ -132,6 +132,9 @@

Roadmap

Planned direction for CSharpDB — organized by timeframe and priority. Reflects the current v3.0.0 state.

+
+ Need the full source guide? The original long-form markdown version is preserved as Roadmap Source Reference. +
diff --git a/www/sitemap.xml b/www/sitemap.xml index 84bbb303..9f4d4101 100644 --- a/www/sitemap.xml +++ b/www/sitemap.xml @@ -12,6 +12,12 @@ monthly 0.7 + + https://csharpdb.com/architecture-reference.html + 2026-04-24 + monthly + 0.5 + https://csharpdb.com/downloads.html 2026-04-20 @@ -24,6 +30,12 @@ monthly 0.7 + + https://csharpdb.com/roadmap-reference.html + 2026-04-24 + monthly + 0.5 + https://csharpdb.com/changelog.html 2026-04-21 @@ -42,6 +54,12 @@ monthly 0.8 + + https://csharpdb.com/docs/getting-started-reference.html + 2026-04-24 + monthly + 0.5 + https://csharpdb.com/docs/admin-ui.html 2026-03-27 @@ -138,12 +156,24 @@ monthly 0.6 + + https://csharpdb.com/docs/performance-reference.html + 2026-04-24 + monthly + 0.5 + https://csharpdb.com/docs/pipelines.html 2026-04-20 monthly 0.6 + + https://csharpdb.com/docs/query-execution-pipeline.html + 2026-04-24 + monthly + 0.5 + https://csharpdb.com/docs/reports.html 2026-04-20 @@ -168,6 +198,12 @@ monthly 0.6 + + https://csharpdb.com/docs/sql-reference.html + 2026-04-24 + monthly + 0.5 + https://csharpdb.com/docs/storage-architecture.html 2026-04-20 @@ -180,6 +216,12 @@ monthly 0.6 + + https://csharpdb.com/docs/storage-engine-reference.html + 2026-04-24 + monthly + 0.5 + https://csharpdb.com/docs/storage-inspector.html 2026-04-20 @@ -270,6 +312,12 @@ monthly 0.6 + + https://csharpdb.com/blog/csharpdb-vs-sqlite-benchmarking-reference.html + 2026-04-24 + monthly + 0.5 + https://csharpdb.com/blog/using-csharpdb-from-multiple-threads.html 2026-04-08 From 102b05b8c32cddbfc55b32e914e5b2eeb1ba6231 Mon Sep 17 00:00:00 2001 From: Maximum Code Date: Sat, 25 Apr 2026 07:02:51 -0700 Subject: [PATCH 02/10] Add v3.4 admin runtime and packaging updates --- .github/workflows/release.yml | 199 ++- CSharpDB.slnx | 15 + RELEASE_NOTES.md | 46 + deploy/daemon/linux/csharpdb-daemon.service | 19 + .../daemon/linux/install-csharpdb-daemon.sh | 117 ++ .../daemon/linux/uninstall-csharpdb-daemon.sh | 38 + deploy/daemon/macos/com.csharpdb.daemon.plist | 36 + .../daemon/macos/install-csharpdb-daemon.sh | 99 ++ .../daemon/macos/uninstall-csharpdb-daemon.sh | 37 + .../windows/install-csharpdb-daemon.ps1 | 122 ++ .../windows/uninstall-csharpdb-daemon.ps1 | 44 + docs/releases/v3.4.0-pr-notes.md | 92 +- global.json | 2 +- samples/README.md | 21 + .../EfCoreProviderSample.csproj | 2 +- .../FulfillmentHubSample.csproj | 29 + samples/fulfillment-hub/Program.cs | 1093 +++++++++++++++++ samples/fulfillment-hub/README.md | 295 +++++ .../imports/marketplace-orders.json | 50 + .../pipelines/low-stock-export.json | 35 + .../pipelines/marketplace-orders-import.json | 92 ++ .../pipelines/supplier-receipts-import.json | 95 ++ samples/fulfillment-hub/procedures.json | 195 +++ samples/fulfillment-hub/queries.sql | 97 ++ samples/fulfillment-hub/saved-queries.json | 22 + samples/fulfillment-hub/schema.sql | 452 +++++++ scripts/Publish-CSharpDbDaemonRelease.ps1 | 200 +++ scripts/README.md | 212 +++- scripts/Start-CSharpDbAdminFormsWeb.ps1 | 292 +++++ .../CSharpDB.Admin.Forms.Web.csproj | 16 + .../Components/App.razor | 17 + .../Components/Layout/MainLayout.razor | 3 + .../Components/Pages/FormRuntime.razor | 9 + .../Components/Pages/Home.razor | 86 ++ .../Components/Routes.razor | 18 + .../FormsHostClientOptionsBuilder.cs | 170 +++ src/CSharpDB.Admin.Forms.Web/Program.cs | 39 + src/CSharpDB.Admin.Forms.Web/README.md | 133 ++ src/CSharpDB.Admin.Forms.Web/_Imports.razor | 10 + src/CSharpDB.Admin.Forms.Web/appsettings.json | 12 + .../wwwroot/css/app.css | 174 +++ .../wwwroot/js/forms-host.js | 64 + .../Designer/PropertyInspector.razor | 2 +- .../Pages/DataEntry.razor | 13 +- .../wwwroot/css/designer.css | 36 + .../Components/Layout/MainLayout.razor | 108 +- .../Components/Shared/DataGrid.razor | 196 ++- .../Components/Shared/SqlEditor.razor | 183 ++- .../Components/Tabs/DataTab.razor | 17 +- .../Components/Tabs/QueryDesignerPanel.razor | 50 +- .../Components/Tabs/QueryTab.razor | 337 +++-- .../Configuration/AdminHostDatabaseOptions.cs | 166 +++ .../Helpers/QueryPagingSqlBuilder.cs | 119 +- .../Models/DataGridFilterMatchMode.cs | 9 + src/CSharpDB.Admin/Program.cs | 31 +- src/CSharpDB.Admin/README.md | 38 +- .../Services/DatabaseClientHolder.cs | 12 +- src/CSharpDB.Admin/appsettings.json | 11 +- src/CSharpDB.Admin/wwwroot/css/app.css | 189 ++- src/CSharpDB.Admin/wwwroot/js/interop.js | 226 +++- src/CSharpDB.Api/CSharpDB.Api.csproj | 4 +- .../CSharpDbRestApiHostExtensions.cs | 118 ++ src/CSharpDB.Api/Program.cs | 59 +- src/CSharpDB.Api/README.md | 19 +- src/CSharpDB.Client/CSharpDB.Client.csproj | 3 +- .../Internal/GrpcTransportClient.cs | 9 + src/CSharpDB.Daemon/CSharpDB.Daemon.csproj | 10 + src/CSharpDB.Daemon/Program.cs | 25 +- src/CSharpDB.Daemon/README.md | 279 ++++- src/CSharpDB.Daemon/appsettings.json | 3 + .../CSharpDB.EntityFrameworkCore.csproj | 4 +- .../CSharpDB.Generators.csproj | 2 +- src/CSharpDB.Mcp/CSharpDB.Mcp.csproj | 2 +- .../Admin/AdminClientOptionsBuilderTests.cs | 120 ++ .../Components/Shared/DataGridTests.cs | 147 +++ .../CSharpDB.Api.Tests.csproj | 2 +- .../CSharpDB.Benchmarks.csproj | 6 +- .../CSharpDB.Daemon.Tests.csproj | 2 +- .../DaemonPackagingAssetsTests.cs | 106 ++ .../CSharpDB.Daemon.Tests/GrpcClientTests.cs | 90 ++ .../CSharpDB.Data.Tests.csproj | 4 +- .../CSharpDB.EntityFrameworkCore.Tests.csproj | 4 +- 82 files changed, 7173 insertions(+), 387 deletions(-) create mode 100644 deploy/daemon/linux/csharpdb-daemon.service create mode 100644 deploy/daemon/linux/install-csharpdb-daemon.sh create mode 100644 deploy/daemon/linux/uninstall-csharpdb-daemon.sh create mode 100644 deploy/daemon/macos/com.csharpdb.daemon.plist create mode 100644 deploy/daemon/macos/install-csharpdb-daemon.sh create mode 100644 deploy/daemon/macos/uninstall-csharpdb-daemon.sh create mode 100644 deploy/daemon/windows/install-csharpdb-daemon.ps1 create mode 100644 deploy/daemon/windows/uninstall-csharpdb-daemon.ps1 create mode 100644 samples/fulfillment-hub/FulfillmentHubSample.csproj create mode 100644 samples/fulfillment-hub/Program.cs create mode 100644 samples/fulfillment-hub/README.md create mode 100644 samples/fulfillment-hub/imports/marketplace-orders.json create mode 100644 samples/fulfillment-hub/pipelines/low-stock-export.json create mode 100644 samples/fulfillment-hub/pipelines/marketplace-orders-import.json create mode 100644 samples/fulfillment-hub/pipelines/supplier-receipts-import.json create mode 100644 samples/fulfillment-hub/procedures.json create mode 100644 samples/fulfillment-hub/queries.sql create mode 100644 samples/fulfillment-hub/saved-queries.json create mode 100644 samples/fulfillment-hub/schema.sql create mode 100644 scripts/Publish-CSharpDbDaemonRelease.ps1 create mode 100644 scripts/Start-CSharpDbAdminFormsWeb.ps1 create mode 100644 src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj create mode 100644 src/CSharpDB.Admin.Forms.Web/Components/App.razor create mode 100644 src/CSharpDB.Admin.Forms.Web/Components/Layout/MainLayout.razor create mode 100644 src/CSharpDB.Admin.Forms.Web/Components/Pages/FormRuntime.razor create mode 100644 src/CSharpDB.Admin.Forms.Web/Components/Pages/Home.razor create mode 100644 src/CSharpDB.Admin.Forms.Web/Components/Routes.razor create mode 100644 src/CSharpDB.Admin.Forms.Web/Configuration/FormsHostClientOptionsBuilder.cs create mode 100644 src/CSharpDB.Admin.Forms.Web/Program.cs create mode 100644 src/CSharpDB.Admin.Forms.Web/README.md create mode 100644 src/CSharpDB.Admin.Forms.Web/_Imports.razor create mode 100644 src/CSharpDB.Admin.Forms.Web/appsettings.json create mode 100644 src/CSharpDB.Admin.Forms.Web/wwwroot/css/app.css create mode 100644 src/CSharpDB.Admin.Forms.Web/wwwroot/js/forms-host.js create mode 100644 src/CSharpDB.Admin/Configuration/AdminHostDatabaseOptions.cs create mode 100644 src/CSharpDB.Admin/Models/DataGridFilterMatchMode.cs create mode 100644 src/CSharpDB.Api/CSharpDbRestApiHostExtensions.cs create mode 100644 tests/CSharpDB.Admin.Forms.Tests/Admin/AdminClientOptionsBuilderTests.cs create mode 100644 tests/CSharpDB.Daemon.Tests/DaemonPackagingAssetsTests.cs diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b9c5ce05..b1ed1091 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -212,9 +212,190 @@ jobs: path: src/CSharpDB.Native/csharpdb.h if-no-files-found: error + daemon-archives: + name: Daemon Archives (${{ matrix.os }} ${{ matrix.rid }}) + needs: build-and-test + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + include: + - os: ubuntu-latest + rid: linux-x64 + extension: tar.gz + executable: CSharpDB.Daemon + - os: windows-latest + rid: win-x64 + extension: zip + executable: CSharpDB.Daemon.exe + - os: macos-latest + rid: osx-arm64 + extension: tar.gz + executable: CSharpDB.Daemon + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Setup .NET + uses: actions/setup-dotnet@v4 + with: + global-json-file: global.json + + - name: Publish daemon archive + shell: pwsh + run: ./scripts/Publish-CSharpDbDaemonRelease.ps1 -Runtime ${{ matrix.rid }} -OutputRoot artifacts/daemon-release + + - name: Verify daemon archive and smoke start + shell: pwsh + run: | + $ErrorActionPreference = 'Stop' + $version = $env:GITHUB_REF_NAME -replace '^v', '' + $archiveName = "csharpdb-daemon-v$version-${{ matrix.rid }}.${{ matrix.extension }}" + $archivePath = Join-Path "artifacts/daemon-release/archives" $archiveName + $checksumsPath = Join-Path "artifacts/daemon-release/archives" "SHA256SUMS.txt" + + if (-not (Test-Path -Path $archivePath)) { + throw "Expected daemon archive was not created: $archivePath" + } + + if (-not (Select-String -Path $checksumsPath -Pattern ([regex]::Escape($archiveName)) -Quiet)) { + throw "Checksum file does not contain $archiveName." + } + + $extractRoot = Join-Path $env:RUNNER_TEMP "csharpdb-daemon-extract" + if (Test-Path -Path $extractRoot) { + Remove-Item -LiteralPath $extractRoot -Recurse -Force + } + New-Item -ItemType Directory -Path $extractRoot | Out-Null + + if ($archiveName.EndsWith('.zip', [StringComparison]::OrdinalIgnoreCase)) { + Expand-Archive -Path $archivePath -DestinationPath $extractRoot -Force + } + else { + & tar -xzf $archivePath -C $extractRoot + if ($LASTEXITCODE -ne 0) { + throw "tar extraction failed with exit code $LASTEXITCODE." + } + } + + $executablePath = Join-Path $extractRoot "${{ matrix.executable }}" + if (-not (Test-Path -Path $executablePath)) { + throw "Expected daemon executable was not found after extraction: $executablePath" + } + + if (-not $IsWindows) { + & chmod +x $executablePath + } + + $outPath = Join-Path $env:RUNNER_TEMP "csharpdb-daemon-smoke.out.log" + $errPath = Join-Path $env:RUNNER_TEMP "csharpdb-daemon-smoke.err.log" + $dbPath = Join-Path $env:RUNNER_TEMP "csharpdb-daemon-smoke.db" + $env:DOTNET_ENVIRONMENT = 'Production' + $env:ASPNETCORE_ENVIRONMENT = 'Production' + $env:ASPNETCORE_URLS = 'http://127.0.0.1:5820' + $env:ConnectionStrings__CSharpDB = "Data Source=$dbPath" + $env:CSharpDB__Daemon__EnableRestApi = 'true' + + $process = Start-Process ` + -FilePath $executablePath ` + -WorkingDirectory $extractRoot ` + -PassThru ` + -RedirectStandardOutput $outPath ` + -RedirectStandardError $errPath + + $info = $null + $deadline = [DateTimeOffset]::UtcNow.AddSeconds(30) + while ([DateTimeOffset]::UtcNow -lt $deadline) { + if ($process.HasExited) { + Write-Host "--- stdout ---" + if (Test-Path $outPath) { Get-Content $outPath } + Write-Host "--- stderr ---" + if (Test-Path $errPath) { Get-Content $errPath } + throw "Daemon smoke process exited early with code $($process.ExitCode)." + } + + try { + $info = Invoke-RestMethod -Uri 'http://127.0.0.1:5820/api/info' -TimeoutSec 2 + break + } + catch { + Start-Sleep -Seconds 1 + } + } + + if ($null -eq $info) { + Write-Host "--- stdout ---" + if (Test-Path $outPath) { Get-Content $outPath } + Write-Host "--- stderr ---" + if (Test-Path $errPath) { Get-Content $errPath } + throw "Daemon REST smoke endpoint did not respond before the timeout." + } + + if ([string]::IsNullOrWhiteSpace($info.dataSource)) { + throw "Daemon REST smoke endpoint returned no dataSource." + } + + $grpcSmokeRoot = Join-Path $env:RUNNER_TEMP "csharpdb-daemon-grpc-smoke" + if (Test-Path -Path $grpcSmokeRoot) { + Remove-Item -LiteralPath $grpcSmokeRoot -Recurse -Force + } + New-Item -ItemType Directory -Path $grpcSmokeRoot | Out-Null + + $clientProjectPath = (Resolve-Path "src/CSharpDB.Client/CSharpDB.Client.csproj").Path + $grpcSmokeProject = Join-Path $grpcSmokeRoot "CSharpDbDaemonGrpcSmoke.csproj" + $grpcSmokeProgram = Join-Path $grpcSmokeRoot "Program.cs" + + @" + + + Exe + net10.0 + enable + enable + + + + + + "@ | Set-Content -Path $grpcSmokeProject -Encoding UTF8 + + @" + using CSharpDB.Client; + + await using var client = CSharpDbClient.Create(new CSharpDbClientOptions + { + Transport = CSharpDbTransport.Grpc, + Endpoint = args[0], + }); + + var info = await client.GetInfoAsync(); + if (string.IsNullOrWhiteSpace(info.DataSource)) + throw new InvalidOperationException("Daemon gRPC smoke returned no data source."); + + Console.WriteLine(info.DataSource); + "@ | Set-Content -Path $grpcSmokeProgram -Encoding UTF8 + + & dotnet run --project $grpcSmokeProject --configuration Release -- 'http://127.0.0.1:5820' + if ($LASTEXITCODE -ne 0) { + throw "Daemon gRPC smoke client failed with exit code $LASTEXITCODE." + } + + $process.Kill($true) + $process.WaitForExit() + + - name: Upload daemon archive + uses: actions/upload-artifact@v4 + with: + name: daemon-${{ matrix.rid }} + path: | + artifacts/daemon-release/archives/*.zip + artifacts/daemon-release/archives/*.tar.gz + if-no-files-found: error + create-release: name: Create GitHub Release - needs: [publish-nuget, native-aot] + needs: [publish-nuget, native-aot, daemon-archives] runs-on: ubuntu-latest permissions: contents: write @@ -236,6 +417,20 @@ jobs: path: artifacts/native merge-multiple: true + - name: Download daemon artifacts + uses: actions/download-artifact@v4 + with: + pattern: daemon-* + path: artifacts/daemon + merge-multiple: true + + - name: Generate daemon checksums + shell: bash + run: | + set -euo pipefail + cd artifacts/daemon + sha256sum csharpdb-daemon-v* > SHA256SUMS.txt + - name: Resolve release notes id: notes shell: bash @@ -270,6 +465,7 @@ jobs: files: | artifacts/nuget/*.nupkg artifacts/native/* + artifacts/daemon/* - name: Create release (auto-generated notes) if: steps.notes.outputs.has_notes == 'false' @@ -280,3 +476,4 @@ jobs: files: | artifacts/nuget/*.nupkg artifacts/native/* + artifacts/daemon/* diff --git a/CSharpDB.slnx b/CSharpDB.slnx index 457d0585..83d8c108 100644 --- a/CSharpDB.slnx +++ b/CSharpDB.slnx @@ -1,7 +1,22 @@ + + + + + + + + + + + + + + + diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 66e401f1..cd51a756 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -41,6 +41,15 @@ client contract. - The Admin SQL query editor now has homegrown guided completions for SQL keywords, table/view selection, select-list columns, qualified alias columns, and stored procedure names without adding a third-party editor dependency. +- The Admin SQL query tab now exposes a visible vertical splitter so the SQL + editor and results pane can be resized when longer queries need more working + space. +- The visual query designer now exposes its own splitter so the generated SQL + preview can be expanded against the results section instead of staying pinned + to a fixed preview height. +- The form designer property inspector now renders the selected control ID with + theme-aware display styling instead of inheriting browser-native readonly + input colors that were hard to read in the dark theme. ### Daemon Service Packaging @@ -91,6 +100,27 @@ client contract. internals, and storage inspector guides after their website versions were audited and verified. +### Samples And Forms Runtime + +- Added `samples/fulfillment-hub`, a runnable end-to-end warehouse and order + fulfillment sample that seeds tables, indexes, views, triggers, procedures, + saved queries, Admin forms, Admin reports, stored pipelines, typed + collections, and a full-text index into one database. +- The Fulfillment Hub sample includes an operational workbook plus a narrative + README walkthrough that teaches the platform through receiving, allocation, + shipment, returns, audit, pipeline, collection, and full-text flows instead + of a flat feature list. +- Added a forms-only runtime host at `src/CSharpDB.Admin.Forms.Web` that lists + stored forms and runs them without exposing the form designer. +- The forms runtime host points at any target CSharpDB database through + `CSharpDB:DataSource`, `ConnectionStrings:CSharpDB`, or `CSharpDB:Endpoint` + and reuses the existing `DataEntry` runtime component from + `CSharpDB.Admin.Forms`. +- `CSharpDB.Admin.Forms` now supports runtime-only hosting through optional + `ShowDesignerButton`, `BackHref`, and `BackLabel` parameters on + `DataEntry.razor`, while keeping the existing Admin studio behavior unchanged + by default. + ### Validation - PowerShell parser validation passed for the daemon release publisher and @@ -112,6 +142,16 @@ client contract. passed for the updated site map. - Repo scan found no remaining references to the deleted duplicated markdown docs. +- `dotnet run --project samples\fulfillment-hub\FulfillmentHubSample.csproj` + completed successfully and seeded `3` forms, `3` reports, `5` procedures, + `5` saved queries, `3` stored pipelines, `2` collections, and the + `fts_ops_playbooks` full-text index into the sample database. +- `dotnet build src\CSharpDB.Admin.Forms.Web\CSharpDB.Admin.Forms.Web.csproj` + completed successfully. +- `dotnet run --project src\CSharpDB.Admin.Forms.Web\CSharpDB.Admin.Forms.Web.csproj -- --urls http://127.0.0.1:5095 --CSharpDB:DataSource=` + started successfully, listed the seeded sample forms at `/`, and served the + runtime-only `orders-workbench` form route at `/forms/orders-workbench` + without the designer action. - `.\scripts\Publish-CSharpDbDaemonRelease.ps1 -Version 3.4.0 -Runtime win-x64 -OutputRoot artifacts\daemon-release-local` created `csharpdb-daemon-v3.4.0-win-x64.zip` and `SHA256SUMS.txt`. - The extracted `win-x64` daemon archive smoke-started successfully, served @@ -125,3 +165,9 @@ client contract. - `dotnet run --project src\CSharpDB.Admin\CSharpDB.Admin.csproj --configuration Release --no-build --no-launch-profile` smoke-started successfully in direct hybrid incremental-durable mode with a temporary database. +- `dotnet build src\CSharpDB.Admin.Forms\CSharpDB.Admin.Forms.csproj` + completed successfully after the form designer property inspector styling + changes. +- `dotnet build src\CSharpDB.Admin\CSharpDB.Admin.csproj -p:BaseOutputPath=artifacts\verify\` + completed successfully after the query tab and visual designer splitter + changes. diff --git a/deploy/daemon/linux/csharpdb-daemon.service b/deploy/daemon/linux/csharpdb-daemon.service new file mode 100644 index 00000000..4e440246 --- /dev/null +++ b/deploy/daemon/linux/csharpdb-daemon.service @@ -0,0 +1,19 @@ +[Unit] +Description=CSharpDB Daemon +After=network-online.target +Wants=network-online.target + +[Service] +Type=notify +WorkingDirectory={{INSTALL_DIR}} +ExecStart={{INSTALL_DIR}}/CSharpDB.Daemon +EnvironmentFile={{ENV_FILE}} +Restart=on-failure +RestartSec=5 +User={{SERVICE_USER}} +Group={{SERVICE_GROUP}} +KillSignal=SIGINT +SyslogIdentifier={{SERVICE_NAME}} + +[Install] +WantedBy=multi-user.target diff --git a/deploy/daemon/linux/install-csharpdb-daemon.sh b/deploy/daemon/linux/install-csharpdb-daemon.sh new file mode 100644 index 00000000..f2506222 --- /dev/null +++ b/deploy/daemon/linux/install-csharpdb-daemon.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +set -euo pipefail + +SERVICE_NAME="csharpdb-daemon" +INSTALL_DIR="/opt/csharpdb-daemon" +DATA_DIR="/var/lib/csharpdb" +URL="http://127.0.0.1:5820" +SERVICE_USER="csharpdb" +SERVICE_GROUP="csharpdb" +SOURCE_DIR="" +FORCE=0 +START=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --service-name) SERVICE_NAME="$2"; shift 2 ;; + --install-dir) INSTALL_DIR="$2"; shift 2 ;; + --data-dir) DATA_DIR="$2"; shift 2 ;; + --url) URL="$2"; shift 2 ;; + --service-user) SERVICE_USER="$2"; shift 2 ;; + --service-group) SERVICE_GROUP="$2"; shift 2 ;; + --source-dir) SOURCE_DIR="$2"; shift 2 ;; + --force) FORCE=1; shift ;; + --start) START=1; shift ;; + -h|--help) + echo "Usage: sudo ./install-csharpdb-daemon.sh [--service-name NAME] [--install-dir PATH] [--data-dir PATH] [--url URL] [--force] [--start]" + exit 0 + ;; + *) echo "Unknown argument: $1" >&2; exit 1 ;; + esac +done + +if [[ "$(id -u)" -ne 0 ]]; then + echo "Installing CSharpDB.Daemon as a systemd service requires root." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +if [[ -z "$SOURCE_DIR" ]]; then + SOURCE_DIR="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +fi + +SERVICE_FILE="/etc/systemd/system/${SERVICE_NAME}.service" +ENV_DIR="/etc/csharpdb-daemon" +ENV_FILE="${ENV_DIR}/${SERVICE_NAME}.env" +DATABASE_PATH="${DATA_DIR}/csharpdb.db" + +if [[ -f "$SERVICE_FILE" && "$FORCE" -ne 1 ]]; then + echo "Service file already exists: $SERVICE_FILE. Re-run with --force to replace it." >&2 + exit 1 +fi + +if ! getent group "$SERVICE_GROUP" >/dev/null 2>&1; then + groupadd --system "$SERVICE_GROUP" +fi + +if ! id -u "$SERVICE_USER" >/dev/null 2>&1; then + useradd --system --gid "$SERVICE_GROUP" --home-dir "$DATA_DIR" --shell /usr/sbin/nologin "$SERVICE_USER" +fi + +if systemctl list-unit-files "${SERVICE_NAME}.service" >/dev/null 2>&1; then + systemctl stop "${SERVICE_NAME}.service" >/dev/null 2>&1 || true +fi + +mkdir -p "$INSTALL_DIR" "$DATA_DIR" "$ENV_DIR" +cp -a "$SOURCE_DIR"/. "$INSTALL_DIR"/ +chmod +x "$INSTALL_DIR/CSharpDB.Daemon" +chown -R "$SERVICE_USER:$SERVICE_GROUP" "$DATA_DIR" + +cat > "$INSTALL_DIR/appsettings.Production.json" < "$ENV_FILE" < "$SERVICE_FILE" + +chmod 0644 "$SERVICE_FILE" +systemctl daemon-reload +systemctl enable "${SERVICE_NAME}.service" + +if [[ "$START" -eq 1 ]]; then + systemctl start "${SERVICE_NAME}.service" +fi + +echo "Installed ${SERVICE_NAME}.service" +echo " Install directory: $INSTALL_DIR" +echo " Data directory: $DATA_DIR" +echo " URL: $URL" diff --git a/deploy/daemon/linux/uninstall-csharpdb-daemon.sh b/deploy/daemon/linux/uninstall-csharpdb-daemon.sh new file mode 100644 index 00000000..0818fe71 --- /dev/null +++ b/deploy/daemon/linux/uninstall-csharpdb-daemon.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +set -euo pipefail + +SERVICE_NAME="csharpdb-daemon" +INSTALL_DIR="/opt/csharpdb-daemon" +REMOVE_INSTALL_DIR=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --service-name) SERVICE_NAME="$2"; shift 2 ;; + --install-dir) INSTALL_DIR="$2"; shift 2 ;; + --remove-install-dir) REMOVE_INSTALL_DIR=1; shift ;; + -h|--help) + echo "Usage: sudo ./uninstall-csharpdb-daemon.sh [--service-name NAME] [--install-dir PATH] [--remove-install-dir]" + exit 0 + ;; + *) echo "Unknown argument: $1" >&2; exit 1 ;; + esac +done + +if [[ "$(id -u)" -ne 0 ]]; then + echo "Uninstalling CSharpDB.Daemon as a systemd service requires root." >&2 + exit 1 +fi + +SERVICE_FILE="/etc/systemd/system/${SERVICE_NAME}.service" +ENV_FILE="/etc/csharpdb-daemon/${SERVICE_NAME}.env" + +systemctl stop "${SERVICE_NAME}.service" >/dev/null 2>&1 || true +systemctl disable "${SERVICE_NAME}.service" >/dev/null 2>&1 || true +rm -f "$SERVICE_FILE" "$ENV_FILE" +systemctl daemon-reload + +if [[ "$REMOVE_INSTALL_DIR" -eq 1 && -d "$INSTALL_DIR" ]]; then + rm -rf "$INSTALL_DIR" +fi + +echo "Uninstalled ${SERVICE_NAME}.service" diff --git a/deploy/daemon/macos/com.csharpdb.daemon.plist b/deploy/daemon/macos/com.csharpdb.daemon.plist new file mode 100644 index 00000000..911b9da9 --- /dev/null +++ b/deploy/daemon/macos/com.csharpdb.daemon.plist @@ -0,0 +1,36 @@ + + + + + Label + {{SERVICE_NAME}} + ProgramArguments + + {{INSTALL_DIR}}/CSharpDB.Daemon + + WorkingDirectory + {{INSTALL_DIR}} + EnvironmentVariables + + DOTNET_ENVIRONMENT + Production + ASPNETCORE_ENVIRONMENT + Production + ASPNETCORE_URLS + {{URL}} + ConnectionStrings__CSharpDB + Data Source={{DATABASE_PATH}} + CSharpDB__Daemon__EnableRestApi + true + + RunAtLoad + + KeepAlive + + StandardOutPath + {{LOG_DIR}}/csharpdb-daemon.out.log + StandardErrorPath + {{LOG_DIR}}/csharpdb-daemon.err.log + + diff --git a/deploy/daemon/macos/install-csharpdb-daemon.sh b/deploy/daemon/macos/install-csharpdb-daemon.sh new file mode 100644 index 00000000..0e548f02 --- /dev/null +++ b/deploy/daemon/macos/install-csharpdb-daemon.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +set -euo pipefail + +SERVICE_NAME="com.csharpdb.daemon" +INSTALL_DIR="/usr/local/lib/csharpdb-daemon" +DATA_DIR="/usr/local/var/csharpdb" +LOG_DIR="/usr/local/var/log" +URL="http://127.0.0.1:5820" +SOURCE_DIR="" +FORCE=0 +START=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --service-name) SERVICE_NAME="$2"; shift 2 ;; + --install-dir) INSTALL_DIR="$2"; shift 2 ;; + --data-dir) DATA_DIR="$2"; shift 2 ;; + --log-dir) LOG_DIR="$2"; shift 2 ;; + --url) URL="$2"; shift 2 ;; + --source-dir) SOURCE_DIR="$2"; shift 2 ;; + --force) FORCE=1; shift ;; + --start) START=1; shift ;; + -h|--help) + echo "Usage: sudo ./install-csharpdb-daemon.sh [--service-name NAME] [--install-dir PATH] [--data-dir PATH] [--url URL] [--force] [--start]" + exit 0 + ;; + *) echo "Unknown argument: $1" >&2; exit 1 ;; + esac +done + +if [[ "$(id -u)" -ne 0 ]]; then + echo "Installing CSharpDB.Daemon as a launchd service requires root." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +if [[ -z "$SOURCE_DIR" ]]; then + SOURCE_DIR="$(cd -- "$SCRIPT_DIR/../.." && pwd)" +fi + +PLIST_PATH="/Library/LaunchDaemons/${SERVICE_NAME}.plist" +DATABASE_PATH="${DATA_DIR}/csharpdb.db" + +if [[ -f "$PLIST_PATH" && "$FORCE" -ne 1 ]]; then + echo "LaunchDaemon already exists: $PLIST_PATH. Re-run with --force to replace it." >&2 + exit 1 +fi + +if [[ -f "$PLIST_PATH" ]]; then + launchctl bootout system "$PLIST_PATH" >/dev/null 2>&1 || true +fi + +mkdir -p "$INSTALL_DIR" "$DATA_DIR" "$LOG_DIR" +cp -a "$SOURCE_DIR"/. "$INSTALL_DIR"/ +chmod +x "$INSTALL_DIR/CSharpDB.Daemon" + +cat > "$INSTALL_DIR/appsettings.Production.json" < "$PLIST_PATH" + +chown root:wheel "$PLIST_PATH" +chmod 0644 "$PLIST_PATH" + +if [[ "$START" -eq 1 ]]; then + launchctl bootstrap system "$PLIST_PATH" + launchctl enable "system/${SERVICE_NAME}" +else + echo "LaunchDaemon installed but not started. Start with: sudo launchctl bootstrap system $PLIST_PATH" +fi + +echo "Installed $SERVICE_NAME" +echo " Install directory: $INSTALL_DIR" +echo " Data directory: $DATA_DIR" +echo " URL: $URL" diff --git a/deploy/daemon/macos/uninstall-csharpdb-daemon.sh b/deploy/daemon/macos/uninstall-csharpdb-daemon.sh new file mode 100644 index 00000000..e1e00791 --- /dev/null +++ b/deploy/daemon/macos/uninstall-csharpdb-daemon.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +set -euo pipefail + +SERVICE_NAME="com.csharpdb.daemon" +INSTALL_DIR="/usr/local/lib/csharpdb-daemon" +REMOVE_INSTALL_DIR=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --service-name) SERVICE_NAME="$2"; shift 2 ;; + --install-dir) INSTALL_DIR="$2"; shift 2 ;; + --remove-install-dir) REMOVE_INSTALL_DIR=1; shift ;; + -h|--help) + echo "Usage: sudo ./uninstall-csharpdb-daemon.sh [--service-name NAME] [--install-dir PATH] [--remove-install-dir]" + exit 0 + ;; + *) echo "Unknown argument: $1" >&2; exit 1 ;; + esac +done + +if [[ "$(id -u)" -ne 0 ]]; then + echo "Uninstalling CSharpDB.Daemon as a launchd service requires root." >&2 + exit 1 +fi + +PLIST_PATH="/Library/LaunchDaemons/${SERVICE_NAME}.plist" + +if [[ -f "$PLIST_PATH" ]]; then + launchctl bootout system "$PLIST_PATH" >/dev/null 2>&1 || true + rm -f "$PLIST_PATH" +fi + +if [[ "$REMOVE_INSTALL_DIR" -eq 1 && -d "$INSTALL_DIR" ]]; then + rm -rf "$INSTALL_DIR" +fi + +echo "Uninstalled $SERVICE_NAME" diff --git a/deploy/daemon/windows/install-csharpdb-daemon.ps1 b/deploy/daemon/windows/install-csharpdb-daemon.ps1 new file mode 100644 index 00000000..0cc4aeab --- /dev/null +++ b/deploy/daemon/windows/install-csharpdb-daemon.ps1 @@ -0,0 +1,122 @@ +<# +.SYNOPSIS +Installs CSharpDB.Daemon as a Windows Service. + +.DESCRIPTION +Copies an extracted daemon release archive to an install directory, writes +appsettings.Production.json, creates a Windows Service, and configures service +environment variables for the database path and bind URL. +#> +[CmdletBinding(SupportsShouldProcess = $true)] +param( + [string]$ServiceName = 'CSharpDBDaemon', + [string]$InstallDirectory = (Join-Path $env:ProgramFiles 'CSharpDB\Daemon'), + [string]$DataDirectory = (Join-Path $env:ProgramData 'CSharpDB'), + [string]$Url = 'http://127.0.0.1:5820', + [string]$SourceDirectory, + [switch]$Force, + [switch]$Start +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +function Assert-Administrator { + $identity = [Security.Principal.WindowsIdentity]::GetCurrent() + $principal = [Security.Principal.WindowsPrincipal]::new($identity) + if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { + throw 'Installing CSharpDB.Daemon as a Windows Service requires an elevated PowerShell session.' + } +} + +function Write-ProductionSettings { + param( + [string]$Path, + [string]$DatabasePath + ) + + $settings = [ordered]@{ + ConnectionStrings = [ordered]@{ + CSharpDB = "Data Source=$DatabasePath" + } + CSharpDB = [ordered]@{ + Daemon = [ordered]@{ + EnableRestApi = $true + } + HostDatabase = [ordered]@{ + OpenMode = 'HybridIncrementalDurable' + ImplicitInsertExecutionMode = 'ConcurrentWriteTransactions' + UseWriteOptimizedPreset = $true + HotTableNames = @() + HotCollectionNames = @() + } + } + } + + $settings | ConvertTo-Json -Depth 10 | Set-Content -Path $Path -Encoding UTF8 +} + +Assert-Administrator + +if ([string]::IsNullOrWhiteSpace($SourceDirectory)) { + $SourceDirectory = (Resolve-Path (Join-Path $PSScriptRoot '..\..')).Path +} +elseif (-not [System.IO.Path]::IsPathRooted($SourceDirectory)) { + $SourceDirectory = (Resolve-Path $SourceDirectory).Path +} + +$existingService = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue +if ($existingService -and -not $Force.IsPresent) { + throw "Service '$ServiceName' already exists. Re-run with -Force to replace it." +} + +if ($PSCmdlet.ShouldProcess($ServiceName, 'Install CSharpDB.Daemon Windows Service')) { + if ($existingService) { + if ($existingService.Status -ne 'Stopped') { + Stop-Service -Name $ServiceName -Force -ErrorAction Stop + $existingService.WaitForStatus('Stopped', [TimeSpan]::FromSeconds(30)) + } + + & sc.exe delete $ServiceName | Out-Null + Start-Sleep -Seconds 1 + } + + New-Item -ItemType Directory -Path $InstallDirectory, $DataDirectory -Force | Out-Null + Copy-Item -Path (Join-Path $SourceDirectory '*') -Destination $InstallDirectory -Recurse -Force + + $exePath = Join-Path $InstallDirectory 'CSharpDB.Daemon.exe' + if (-not (Test-Path -Path $exePath)) { + throw "Could not find daemon executable after copy: $exePath" + } + + $databasePath = Join-Path $DataDirectory 'csharpdb.db' + Write-ProductionSettings -Path (Join-Path $InstallDirectory 'appsettings.Production.json') -DatabasePath $databasePath + + New-Service ` + -Name $ServiceName ` + -BinaryPathName "`"$exePath`"" ` + -DisplayName 'CSharpDB Daemon' ` + -Description 'CSharpDB remote daemon service.' ` + -StartupType Automatic | Out-Null + + $serviceKey = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName" + $environment = @( + 'DOTNET_ENVIRONMENT=Production', + 'ASPNETCORE_ENVIRONMENT=Production', + "ASPNETCORE_URLS=$Url", + "ConnectionStrings__CSharpDB=Data Source=$databasePath", + 'CSharpDB__Daemon__EnableRestApi=true' + ) + New-ItemProperty -Path $serviceKey -Name Environment -PropertyType MultiString -Value $environment -Force | Out-Null + + & sc.exe failure $ServiceName reset= 60 actions= restart/5000/restart/10000/''/0 | Out-Null + + if ($Start.IsPresent) { + Start-Service -Name $ServiceName + } + + Write-Host "Installed service '$ServiceName'." + Write-Host " Install directory: $InstallDirectory" + Write-Host " Data directory: $DataDirectory" + Write-Host " URL: $Url" +} diff --git a/deploy/daemon/windows/uninstall-csharpdb-daemon.ps1 b/deploy/daemon/windows/uninstall-csharpdb-daemon.ps1 new file mode 100644 index 00000000..b01f5768 --- /dev/null +++ b/deploy/daemon/windows/uninstall-csharpdb-daemon.ps1 @@ -0,0 +1,44 @@ +<# +.SYNOPSIS +Uninstalls the CSharpDB.Daemon Windows Service. +#> +[CmdletBinding(SupportsShouldProcess = $true)] +param( + [string]$ServiceName = 'CSharpDBDaemon', + [string]$InstallDirectory = (Join-Path $env:ProgramFiles 'CSharpDB\Daemon'), + [switch]$RemoveInstallDirectory +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +function Assert-Administrator { + $identity = [Security.Principal.WindowsIdentity]::GetCurrent() + $principal = [Security.Principal.WindowsPrincipal]::new($identity) + if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { + throw 'Uninstalling CSharpDB.Daemon as a Windows Service requires an elevated PowerShell session.' + } +} + +Assert-Administrator + +if ($PSCmdlet.ShouldProcess($ServiceName, 'Uninstall CSharpDB.Daemon Windows Service')) { + $service = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue + if ($service) { + if ($service.Status -ne 'Stopped') { + Stop-Service -Name $ServiceName -Force -ErrorAction Stop + $service.WaitForStatus('Stopped', [TimeSpan]::FromSeconds(30)) + } + + & sc.exe delete $ServiceName | Out-Null + Write-Host "Deleted service '$ServiceName'." + } + else { + Write-Host "Service '$ServiceName' was not installed." + } + + if ($RemoveInstallDirectory.IsPresent -and (Test-Path -Path $InstallDirectory)) { + Remove-Item -LiteralPath $InstallDirectory -Recurse -Force + Write-Host "Removed install directory: $InstallDirectory" + } +} diff --git a/docs/releases/v3.4.0-pr-notes.md b/docs/releases/v3.4.0-pr-notes.md index 2ef8c881..4f693b7a 100644 --- a/docs/releases/v3.4.0-pr-notes.md +++ b/docs/releases/v3.4.0-pr-notes.md @@ -1,25 +1,37 @@ ## Summary -This PR updates the `v3.4.0` docs/site work so the website is now the -authoritative published home for the verified high-confidence docs set, while -still preserving the original long-form markdown content for the source-heavy -guides that had only been partially migrated before. - -The website work adds companion source-reference HTML pages for the original -architecture, getting-started, performance, query-execution-pipeline, SQL, -storage-engine, roadmap, and SQLite benchmarking markdown guides. The shorter -curated pages remain in place, but now link to the full source references when -readers need the original complete material. The docs cleanup also removes the -duplicate markdown copies of the CLI, REST API, MCP server, internals, and -storage inspector guides after their website versions were verified. - -This PR also adds a new blog post covering the C# launcher pattern for -`CSharpDB.Admin` and keeps the post wired into the blog index and sitemap. +This PR broadens the `v3.4.0` website/docs cleanup into a fuller sample and +runtime-hosting pass. + +On the website side, the repo now preserves the source-heavy markdown guides +under companion source-reference HTML pages and removes only the duplicated +markdown files that were verified as safe to delete. The site work also adds a +new blog post covering the C# launcher pattern for `CSharpDB.Admin`. + +On the sample side, this PR adds `samples/fulfillment-hub`, a runnable +operations showcase that seeds relational schema, views, triggers, procedures, +saved queries, Admin forms, Admin reports, stored pipelines, typed collections, +and a full-text index into one sample database. The sample README is now a +guided story walkthrough so users can learn the platform by running through a +day in warehouse operations instead of reading a flat checklist. + +This PR also adds `src/CSharpDB.Admin.Forms.Web`, a forms-only web host that +lists stored forms from a target database and runs them without exposing design +mode. The new host reuses the existing `DataEntry` runtime component from +`CSharpDB.Admin.Forms`, with small opt-in parameters added so runtime-only hosts +can hide the designer action and provide a back link without changing default +Admin studio behavior. + +The current range also includes a small Admin UI polish pass: the form designer +property inspector now renders selected control IDs with theme-aware styling, +the SQL query tab has a visible resizable splitter between editor and results, +and the visual query designer has a matching splitter for the generated SQL +preview versus results. ## Type of Change -- [ ] Bug fix -- [ ] New feature +- [x] Bug fix +- [x] New feature - [ ] Breaking change - [x] Documentation update - [x] Refactor / maintenance @@ -27,14 +39,24 @@ This PR also adds a new blog post covering the C# launcher pattern for ## Related Issues -No issue numbers were linked for this docs/site cleanup. Included work in this -PR: +No issue numbers were linked for this docs/site, sample, and forms-runtime +work. Included work in this PR: - audited the current `docs/` markdown tree against the published `www` site - added source-reference HTML pages for the partially migrated long-form docs - linked the curated pages to those full references and updated the sitemap - removed high-confidence duplicated markdown files after link cleanup - added the `CSharpDB.Admin` C# launcher blog post to the website +- added the `samples/fulfillment-hub` runnable sample with forms, reports, + procedures, saved queries, pipelines, collections, and full-text search +- rewrote the Fulfillment Hub README into a guided operator walkthrough +- added the `CSharpDB.Admin.Forms.Web` runtime-only forms host +- updated `CSharpDB.Admin.Forms` runtime entry so the shared form runner can be + hosted without a visible designer action +- fixed the form designer property inspector control ID rendering so it no + longer relies on hard-to-read readonly input colors in dark theme +- added a resizable SQL editor/results splitter in the query tab +- added a resizable SQL preview/results splitter in the visual query designer ## Testing @@ -61,6 +83,31 @@ Validation performed for this PR: - repo scan found no remaining references to the deleted duplicated markdown files (`cli.md`, `rest-api.md`, `mcp-server.md`, `internals.md`, `storage-inspector.md`) +- `dotnet run --project samples\fulfillment-hub\FulfillmentHubSample.csproj` + completed successfully and seeded the sample database with: + - `3` forms + - `3` reports + - `5` procedures + - `5` saved queries + - `3` stored pipelines and `3` successful pipeline runs + - `2` collections + - the `fts_ops_playbooks` full-text index +- `dotnet build src\CSharpDB.Admin.Forms.Web\CSharpDB.Admin.Forms.Web.csproj` + completed successfully +- `dotnet build src\CSharpDB.Admin.Forms\CSharpDB.Admin.Forms.csproj` + completed successfully after the property inspector styling change +- `dotnet build src\CSharpDB.Admin\CSharpDB.Admin.csproj -p:BaseOutputPath=C:\Users\maxim\source\Code\CSharpDB\artifacts\verify\` + completed successfully after the query tab and visual designer splitter + changes +- `dotnet run --project src\CSharpDB.Admin.Forms.Web\CSharpDB.Admin.Forms.Web.csproj -- --urls http://127.0.0.1:5095 --CSharpDB:DataSource=` + started successfully +- HTTP verification against the new forms host confirmed: + - `/` lists the seeded forms `Order Workbench`, `Purchase Order Receiving`, + and `Return Intake` + - `/forms/orders-workbench` returns `200` + - the runtime page still shows standard form actions such as `Save` and + `Delete` + - the runtime page does not expose the `Edit Form` designer action ## Checklist @@ -80,3 +127,10 @@ Validation performed for this PR: - Only the high-confidence duplicated markdown files were removed in this pass. The remaining markdown files still need their own migration/verification work before they are safe to delete. +- The forms-only web host intentionally reuses the shared `DataEntry` runtime + component rather than forking it into a separate codepath. The only shared + component changes are opt-in runtime-hosting parameters; existing Admin + behavior stays unchanged by default. +- The latest Admin UI fixes are narrow and local: one theme/readability fix in + the form designer property inspector, one splitter for SQL mode, and one + splitter for visual designer mode. diff --git a/global.json b/global.json index e15758c6..2d614ae4 100644 --- a/global.json +++ b/global.json @@ -1,6 +1,6 @@ { "sdk": { - "version": "10.0.202", + "version": "10.0.203", "rollForward": "latestPatch" } } diff --git a/samples/README.md b/samples/README.md index bbb32933..8d38f2d6 100644 --- a/samples/README.md +++ b/samples/README.md @@ -14,6 +14,7 @@ The SQL dataset samples use a conventional layout with `schema.sql` for setup, ` | `procurement-analytics/` | Query expansion + planner stats workbook | `schema.sql`, `procedures.json`, `queries.sql` | | `feature-tour/` | Northstar Field Services | `schema.sql`, `procedures.json`, `queries.sql` | | `platform-showcase/` | Broad relational feature tour + optional API demo | `schema.sql`, `procedures.json`, `queries.sql`, `.csproj`, `Program.cs` | +| `fulfillment-hub/` | Full-stack operations showcase with forms, reports, pipelines, collections, and full-text search | `schema.sql`, `procedures.json`, `saved-queries.json`, `queries.sql`, `pipelines/`, `imports/`, `.csproj`, `Program.cs`, `README.md` | | `csv-bulk-import/` | Runnable CSV-to-table bulk ingest walkthrough | `.csproj`, `Program.cs`, `README.md`, `events.csv` | | `collection-indexing/` | Runnable `Collection` indexing walkthrough | `.csproj`, `Program.cs`, `README.md` | | `generated-collections/` | Runnable source-generated collection fast-path walkthrough | `.csproj`, `Program.cs`, `README.md` | @@ -79,6 +80,18 @@ Root-level helpers: - Domain: subscriptions, orders, support operations, inventory, knowledge articles, and dashboard presets - Good for: foreign keys, collations, unique + composite indexes, views, triggers, `IDENTITY` audit rows, joins, CTEs, subqueries, set operations, `TEXT(...)`, `ANALYZE`, full-text search, and `Collection` +### Fulfillment Hub + +- SQL: [schema.sql](fulfillment-hub/schema.sql) +- Procedures: [procedures.json](fulfillment-hub/procedures.json) +- Saved Queries: [saved-queries.json](fulfillment-hub/saved-queries.json) +- Queries: [queries.sql](fulfillment-hub/queries.sql) +- Pipelines: [pipelines/](fulfillment-hub/pipelines) +- Demo: [FulfillmentHubSample.csproj](fulfillment-hub/FulfillmentHubSample.csproj) +- Docs: [README.md](fulfillment-hub/README.md) +- Domain: local-first warehouse, order fulfillment, receiving, shipment, and returns operations +- Good for: end-to-end admin workflows, stored procedures, saved queries, reports, forms, CSV/JSON pipelines, typed collections with path indexes, and full-text search + ### Bulk Import / CSV To Table - Project: [CsvBulkImportSample.csproj](csv-bulk-import/CsvBulkImportSample.csproj) @@ -218,6 +231,14 @@ dotnet run --project samples/efcore-provider/EfCoreProviderSample.csproj This sample is the quickest way to validate the embedded EF Core provider, `UseCSharpDb(...)`, navigation loading, and the design-time context setup used by `dotnet ef`. +### Option 9: Run the Fulfillment Hub Sample + +```bash +dotnet run --project samples/fulfillment-hub/FulfillmentHubSample.csproj +``` + +This is the broadest runnable operations sample in the repo. It rebuilds a fresh database, seeds the relational schema and snapshot data, stores procedures and saved queries, persists admin forms and reports, saves and runs pipelines, seeds collections, and creates a full-text index over operational playbooks. + ## v2.2.0 API Examples The SQL samples above cover the relational surface. The following snippets show v2.2.0 features that are accessed through the `CSharpDB.Client` and `CSharpDB.Engine` C# APIs. diff --git a/samples/efcore-provider/EfCoreProviderSample.csproj b/samples/efcore-provider/EfCoreProviderSample.csproj index 314a1762..69b073aa 100644 --- a/samples/efcore-provider/EfCoreProviderSample.csproj +++ b/samples/efcore-provider/EfCoreProviderSample.csproj @@ -12,7 +12,7 @@ - + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/samples/fulfillment-hub/FulfillmentHubSample.csproj b/samples/fulfillment-hub/FulfillmentHubSample.csproj new file mode 100644 index 00000000..5c2c4466 --- /dev/null +++ b/samples/fulfillment-hub/FulfillmentHubSample.csproj @@ -0,0 +1,29 @@ + + + + Exe + net10.0 + enable + enable + + + + + + + + + + + + + + + + + + + + + + diff --git a/samples/fulfillment-hub/Program.cs b/samples/fulfillment-hub/Program.cs new file mode 100644 index 00000000..5a74a517 --- /dev/null +++ b/samples/fulfillment-hub/Program.cs @@ -0,0 +1,1093 @@ +using System.Globalization; +using System.Text.Json; +using CSharpDB.Admin.Forms.Models; +using CSharpDB.Admin.Forms.Services; +using CSharpDB.Admin.Reports.Models; +using CSharpDB.Admin.Reports.Services; +using CSharpDB.Client; +using CSharpDB.Client.Models; +using CSharpDB.Client.Pipelines; +using CSharpDB.Engine; +using CSharpDB.Pipelines.Models; +using CSharpDB.Pipelines.Serialization; +using CSharpDB.Sql; +using Forms = CSharpDB.Admin.Forms.Models; +using Reports = CSharpDB.Admin.Reports.Models; + +var jsonOptions = new JsonSerializerOptions +{ + PropertyNameCaseInsensitive = true, +}; + +string sampleDirectory = AppContext.BaseDirectory; +string schemaPath = Path.Combine(sampleDirectory, "schema.sql"); +string proceduresPath = Path.Combine(sampleDirectory, "procedures.json"); +string savedQueriesPath = Path.Combine(sampleDirectory, "saved-queries.json"); +string pipelinesDirectory = Path.Combine(sampleDirectory, "pipelines"); +string outputDirectory = Path.Combine(sampleDirectory, "generated-output"); +string dbPath = Path.Combine(sampleDirectory, "fulfillment-hub-demo.db"); + +PrepareFreshRun(dbPath, outputDirectory); + +FullTextSeedSummary fullTextSummary; +CollectionSeedSummary collectionSummary; + +await using (Database db = await Database.OpenAsync(dbPath)) +{ + await ExecuteSchemaAsync(db, schemaPath); + await db.EnsureFullTextIndexAsync("fts_ops_playbooks", "ops_playbooks", ["title", "body"]); + fullTextSummary = await CaptureFullTextSummaryAsync(db); + collectionSummary = await SeedCollectionsAsync(db); +} + +await using ICSharpDbClient client = CSharpDbClient.Create(new CSharpDbClientOptions +{ + Transport = CSharpDbTransport.Direct, + DataSource = dbPath, +}); + +await ImportProceduresAsync(client, proceduresPath, jsonOptions); +await ImportSavedQueriesAsync(client, savedQueriesPath, jsonOptions); + +DbFormRepository formRepository = new(client); +DbSchemaProvider schemaProvider = new(client); +await SeedFormsAsync(formRepository, schemaProvider); + +DbReportRepository reportRepository = new(client); +DbReportSourceProvider reportSourceProvider = new(client); +await SeedReportsAsync(reportRepository, reportSourceProvider); + +CSharpDbPipelineCatalogClient pipelineCatalog = new(client); +PipelineSeedSummary pipelineSummary = await SeedPipelinesAsync(pipelineCatalog, pipelinesDirectory, sampleDirectory, outputDirectory); +ProcedureExecutionResult procedureSmokeTest = await client.ExecuteProcedureAsync("RefreshOperationalStats", new Dictionary()); +if (!procedureSmokeTest.Succeeded) + throw new InvalidOperationException($"Procedure smoke test failed: {procedureSmokeTest.Error}"); + +IReadOnlyList procedures = await client.GetProceduresAsync(); +IReadOnlyList savedQueries = await client.GetSavedQueriesAsync(); +IReadOnlyList forms = await formRepository.ListAsync(); +IReadOnlyList reports = await reportRepository.ListAsync(); +IReadOnlyList storedPipelines = await pipelineCatalog.ListPipelinesAsync(); +IReadOnlyList pipelineRuns = await pipelineCatalog.ListRunsAsync(); + +Console.WriteLine("Fulfillment Hub"); +Console.WriteLine(); +Console.WriteLine($"Database: {dbPath}"); +Console.WriteLine($"Forms: {forms.Count} | Reports: {reports.Count} | Procedures: {procedures.Count} | Saved queries: {savedQueries.Count}"); +Console.WriteLine($"Stored pipelines: {storedPipelines.Count} | Pipeline runs: {pipelineRuns.Count}"); +Console.WriteLine($"Procedure smoke test: {procedureSmokeTest.ProcedureName} -> {procedureSmokeTest.Statements.Count} statements"); +Console.WriteLine($"Collections: scanner_sessions={collectionSummary.ScannerSessionCount}, webhook_archive={collectionSummary.WebhookCount}"); +Console.WriteLine(); + +Console.WriteLine("Top open order queue rows:"); +await PrintQueryAsync(client, """ + SELECT order_number, customer_name, warehouse_code, order_status, required_ship_date + FROM order_fulfillment_board + WHERE order_status IN ('released', 'allocated', 'picking') + ORDER BY required_ship_date, priority_code DESC, order_number + LIMIT 4; + """); + +Console.WriteLine(); +Console.WriteLine("Low-stock watch:"); +await PrintQueryAsync(client, """ + SELECT warehouse_code, sku, available_qty, inbound_qty, reorder_point, shortage_qty + FROM low_stock_watch + WHERE shortage_qty > 0 + ORDER BY shortage_qty DESC, warehouse_code, sku + LIMIT 6; + """); + +Console.WriteLine(); +Console.WriteLine("Full-text hits for 'partial receipt':"); +if (fullTextSummary.Hits.Count == 0) +{ + Console.WriteLine(" (no hits)"); +} +else +{ + foreach (string line in fullTextSummary.Hits) + Console.WriteLine($" {line}"); +} + +Console.WriteLine(); +Console.WriteLine("Collection lookups:"); +Console.WriteLine($" scanner_sessions by CurrentWave.OrderNumber=SO-7005 -> {string.Join(", ", collectionSummary.ScannerSessionMatches)}"); +Console.WriteLine($" webhook_archive by $.tags[]=cold-chain -> {string.Join(", ", collectionSummary.WebhookTagMatches)}"); + +Console.WriteLine(); +Console.WriteLine("Pipeline outputs:"); +foreach (string line in pipelineSummary.RunSummaries) + Console.WriteLine($" {line}"); +Console.WriteLine($" Export file: {pipelineSummary.ExportPath}"); + +Console.WriteLine(); +Console.WriteLine("Seeded forms:"); +foreach (Forms.FormDefinition form in forms.OrderBy(item => item.Name, StringComparer.OrdinalIgnoreCase)) + Console.WriteLine($" {form.FormId} -> {form.Name} ({form.TableName})"); + +Console.WriteLine(); +Console.WriteLine("Seeded reports:"); +foreach (Reports.ReportDefinition report in reports.OrderBy(item => item.Name, StringComparer.OrdinalIgnoreCase)) + Console.WriteLine($" {report.ReportId} -> {report.Name} ({report.Source.Kind}:{report.Source.Name})"); + +static void PrepareFreshRun(string dbPath, string outputDirectory) +{ + foreach (string path in new[] { dbPath, $"{dbPath}-wal", $"{dbPath}-shm" }) + { + if (File.Exists(path)) + File.Delete(path); + } + + if (Directory.Exists(outputDirectory)) + Directory.Delete(outputDirectory, recursive: true); + + Directory.CreateDirectory(outputDirectory); +} + +static async Task ExecuteSchemaAsync(Database db, string schemaPath) +{ + string script = await File.ReadAllTextAsync(schemaPath); + foreach (string statement in SqlScriptSplitter.SplitExecutableStatements(script)) + await db.ExecuteAsync(statement); +} + +static async Task CaptureFullTextSummaryAsync(Database db) +{ + IReadOnlyList hits = await db.SearchAsync("fts_ops_playbooks", "partial receipt"); + var lines = new List(hits.Count); + + foreach (FullTextSearchHit hit in hits.Take(3)) + { + await using var titleResult = await db.ExecuteAsync($""" + SELECT title + FROM ops_playbooks + WHERE id = {hit.RowId}; + """); + + var rows = await titleResult.ToListAsync(); + string title = rows.Count > 0 ? rows[0][0].AsText : $"Playbook {hit.RowId}"; + lines.Add($"row={hit.RowId} | score={hit.Score:F2} | {title}"); + } + + return new FullTextSeedSummary(lines); +} + +static async Task SeedCollectionsAsync(Database db) +{ + Collection scannerSessions = await db.GetCollectionAsync("scanner_sessions"); + await scannerSessions.PutAsync("session:sea:wave-a", new ScannerSessionDocument( + "scanner-01", + "SEA-FC", + "Ava Cole", + "wave-a", + new ScannerWaveState("SO-7005", "picking", "2026-04-24T10:15:00Z"), + ["BOT-220", "TRJ-100"], + ["night-shift", "priority-lane"], + "2026-04-24T10:17:00Z")); + await scannerSessions.PutAsync("session:den:receiving", new ScannerSessionDocument( + "scanner-07", + "DEN-DC", + "Marcus Lin", + "receiving", + new ScannerWaveState("PO-9002", "receiving", "2026-04-24T09:42:00Z"), + ["MPR-510", "CCS-810"], + ["receiving", "cold-chain"], + "2026-04-24T09:50:00Z")); + await scannerSessions.EnsureIndexAsync(item => item.WarehouseCode); + await scannerSessions.EnsureIndexAsync("CurrentWave.OrderNumber"); + await scannerSessions.EnsureIndexAsync("$.tags[]"); + + Collection webhookArchive = await db.GetCollectionAsync("webhook_archive"); + await webhookArchive.PutAsync("webhook:carrier:1", new WebhookArchiveDocument( + "carrier-webhook", + "shipment.scan", + "2026-04-24T08:14:00Z", + new WebhookHeaders("SO-7003", "SHP-8001"), + ["carriers", "tracking"], + """{"tracking":"1Z999AA10123456784","status":"delivered"}""")); + await webhookArchive.PutAsync("webhook:marketplace:2", new WebhookArchiveDocument( + "marketplace-webhook", + "inventory.adjustment", + "2026-04-24T08:42:00Z", + new WebhookHeaders("SO-7005", null), + ["marketplace", "cold-chain"], + """{"sku":"CCS-810","reason":"safety-stock-floor"}""")); + await webhookArchive.EnsureIndexAsync(item => item.Provider); + await webhookArchive.EnsureIndexAsync("Headers.OrderNumber"); + await webhookArchive.EnsureIndexAsync("$.tags[]"); + + var scannerMatches = new List(); + await foreach (var match in scannerSessions.FindByPathAsync("CurrentWave.OrderNumber", "SO-7005")) + scannerMatches.Add(match.Key); + + var webhookMatches = new List(); + await foreach (var match in webhookArchive.FindByPathAsync("$.tags[]", "cold-chain")) + webhookMatches.Add(match.Key); + + return new CollectionSeedSummary( + ScannerSessionCount: (int)await scannerSessions.CountAsync(), + WebhookCount: (int)await webhookArchive.CountAsync(), + ScannerSessionMatches: scannerMatches, + WebhookTagMatches: webhookMatches); +} + +static async Task ImportProceduresAsync(ICSharpDbClient client, string proceduresPath, JsonSerializerOptions jsonOptions) +{ + string json = await File.ReadAllTextAsync(proceduresPath); + List? fileDefinitions = JsonSerializer.Deserialize>(json, jsonOptions); + if (fileDefinitions is null) + throw new InvalidOperationException("Procedure file did not deserialize."); + + foreach (StoredProcedureFile fileDefinition in fileDefinitions) + { + ProcedureDefinition definition = new() + { + Name = fileDefinition.Name, + BodySql = fileDefinition.BodySql, + Description = fileDefinition.Description, + IsEnabled = fileDefinition.IsEnabled, + Parameters = fileDefinition.Parameters.Select(parameter => new ProcedureParameterDefinition + { + Name = parameter.Name, + Type = Enum.Parse(parameter.Type, ignoreCase: true), + Required = parameter.Required, + Default = ConvertJsonValue(parameter.Default), + Description = parameter.Description, + }).ToArray(), + }; + + ProcedureDefinition? existing = await client.GetProcedureAsync(definition.Name); + if (existing is null) + await client.CreateProcedureAsync(definition); + else + await client.UpdateProcedureAsync(definition.Name, definition); + } +} + +static async Task ImportSavedQueriesAsync(ICSharpDbClient client, string savedQueriesPath, JsonSerializerOptions jsonOptions) +{ + string json = await File.ReadAllTextAsync(savedQueriesPath); + List? queries = JsonSerializer.Deserialize>(json, jsonOptions); + if (queries is null) + throw new InvalidOperationException("Saved query file did not deserialize."); + + foreach (SavedQueryFile query in queries) + await client.UpsertSavedQueryAsync(query.Name, query.SqlText); +} + +static async Task SeedFormsAsync(DbFormRepository repository, DbSchemaProvider schemaProvider) +{ + Forms.FormTableDefinition orders = await RequireTableAsync(schemaProvider, "orders"); + Forms.FormTableDefinition purchaseOrders = await RequireTableAsync(schemaProvider, "purchase_orders"); + Forms.FormTableDefinition returns = await RequireTableAsync(schemaProvider, "returns"); + + await repository.CreateAsync(CreateOrderWorkbenchForm(orders)); + await repository.CreateAsync(CreatePurchaseOrderReceivingForm(purchaseOrders)); + await repository.CreateAsync(CreateReturnIntakeForm(returns)); +} + +static async Task SeedReportsAsync(DbReportRepository repository, DbReportSourceProvider sourceProvider) +{ + Reports.ReportSourceDefinition shipmentManifestSource = await RequireReportSourceAsync(sourceProvider, ReportSourceKind.View, "shipment_manifest_report_source"); + Reports.ReportSourceDefinition lowStockSource = await RequireReportSourceAsync(sourceProvider, ReportSourceKind.View, "low_stock_watch"); + Reports.ReportSourceDefinition openOrderSource = await RequireReportSourceAsync(sourceProvider, ReportSourceKind.SavedQuery, "Open Order Queue"); + + await repository.CreateAsync(CreateShipmentManifestReport(shipmentManifestSource)); + await repository.CreateAsync(CreateLowStockReport(lowStockSource)); + await repository.CreateAsync(CreateOpenOrderQueueReport(openOrderSource)); +} + +static async Task SeedPipelinesAsync( + CSharpDbPipelineCatalogClient pipelineCatalog, + string pipelinesDirectory, + string sampleDirectory, + string outputDirectory) +{ + string[] pipelineFiles = + [ + Path.Combine(pipelinesDirectory, "supplier-receipts-import.json"), + Path.Combine(pipelinesDirectory, "marketplace-orders-import.json"), + Path.Combine(pipelinesDirectory, "low-stock-export.json"), + ]; + + var runSummaries = new List(pipelineFiles.Length); + foreach (string pipelineFile in pipelineFiles) + { + PipelinePackageDefinition filePackage = await PipelinePackageSerializer.LoadFromFileAsync(pipelineFile); + PipelinePackageDefinition package = ResolvePipelinePaths(filePackage, sampleDirectory, outputDirectory); + await pipelineCatalog.SavePipelineAsync(package); + PipelineRunResult run = await pipelineCatalog.RunStoredPipelineAsync(package.Name); + runSummaries.Add($"{package.Name} -> {run.Status} | rowsRead={run.Metrics.RowsRead} | rowsWritten={run.Metrics.RowsWritten} | rejects={run.Metrics.RowsRejected}"); + } + + return new PipelineSeedSummary( + RunSummaries: runSummaries, + ExportPath: Path.Combine(outputDirectory, "low-stock-watch.csv")); +} + +static PipelinePackageDefinition ResolvePipelinePaths(PipelinePackageDefinition package, string sampleDirectory, string outputDirectory) +{ + string ResolvePath(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + return value ?? string.Empty; + + return value + .Replace("__SAMPLE_DIR__", sampleDirectory, StringComparison.Ordinal) + .Replace("__OUTPUT_DIR__", outputDirectory, StringComparison.Ordinal); + } + + return new PipelinePackageDefinition + { + Name = package.Name, + Version = package.Version, + Description = package.Description, + Source = new PipelineSourceDefinition + { + Kind = package.Source.Kind, + Path = ResolvePath(package.Source.Path), + ConnectionString = package.Source.ConnectionString, + TableName = package.Source.TableName, + QueryText = package.Source.QueryText, + HasHeaderRow = package.Source.HasHeaderRow, + }, + Transforms = package.Transforms.Select(transform => new PipelineTransformDefinition + { + Kind = transform.Kind, + SelectColumns = transform.SelectColumns?.ToArray(), + RenameMappings = transform.RenameMappings?.Select(mapping => new PipelineRenameMapping + { + Source = mapping.Source, + Target = mapping.Target, + }).ToArray(), + CastMappings = transform.CastMappings?.Select(mapping => new PipelineCastMapping + { + Column = mapping.Column, + TargetType = mapping.TargetType, + }).ToArray(), + FilterExpression = transform.FilterExpression, + DerivedColumns = transform.DerivedColumns?.Select(column => new PipelineDerivedColumn + { + Name = column.Name, + Expression = column.Expression, + }).ToArray(), + DeduplicateKeys = transform.DeduplicateKeys?.ToArray(), + }).ToArray(), + Destination = new PipelineDestinationDefinition + { + Kind = package.Destination.Kind, + Path = ResolvePath(package.Destination.Path), + ConnectionString = package.Destination.ConnectionString, + TableName = package.Destination.TableName, + Overwrite = package.Destination.Overwrite, + }, + Options = new PipelineExecutionOptions + { + BatchSize = package.Options.BatchSize, + ErrorMode = package.Options.ErrorMode, + CheckpointInterval = package.Options.CheckpointInterval, + MaxRejects = package.Options.MaxRejects, + }, + Incremental = package.Incremental is null + ? null + : new PipelineIncrementalOptions + { + WatermarkColumn = package.Incremental.WatermarkColumn, + LastProcessedValue = package.Incremental.LastProcessedValue, + }, + }; +} + +static async Task PrintQueryAsync(ICSharpDbClient client, string sql) +{ + SqlExecutionResult result = await client.ExecuteSqlAsync(sql); + if (!string.IsNullOrWhiteSpace(result.Error)) + throw new InvalidOperationException(result.Error); + + if (result.Rows is null || result.Rows.Count == 0) + { + Console.WriteLine(" (no rows)"); + return; + } + + foreach (object?[] row in result.Rows) + Console.WriteLine($" {string.Join(" | ", row.Select(FormatValue))}"); +} + +static string FormatValue(object? value) +{ + if (value is null) + return "NULL"; + + return value switch + { + double number => number.ToString("F2", CultureInfo.InvariantCulture), + float number => number.ToString("F2", CultureInfo.InvariantCulture), + decimal number => number.ToString("F2", CultureInfo.InvariantCulture), + _ => Convert.ToString(value, CultureInfo.InvariantCulture) ?? string.Empty, + }; +} + +static object? ConvertJsonValue(JsonElement? element) +{ + if (!element.HasValue) + return null; + + JsonElement value = element.Value; + return value.ValueKind switch + { + JsonValueKind.Null or JsonValueKind.Undefined => null, + JsonValueKind.True => true, + JsonValueKind.False => false, + JsonValueKind.Number when value.TryGetInt64(out long int64) => int64, + JsonValueKind.Number => value.GetDouble(), + JsonValueKind.String => value.GetString(), + _ => value.ToString(), + }; +} + +static async Task RequireTableAsync(DbSchemaProvider schemaProvider, string tableName) +{ + return await schemaProvider.GetTableDefinitionAsync(tableName) + ?? throw new InvalidOperationException($"Form table '{tableName}' was not found."); +} + +static async Task RequireReportSourceAsync(DbReportSourceProvider sourceProvider, ReportSourceKind kind, string name) +{ + return await sourceProvider.GetSourceDefinitionAsync(new Reports.ReportSourceReference(kind, name)) + ?? throw new InvalidOperationException($"Report source '{kind}:{name}' was not found."); +} + +static Forms.FormDefinition CreateOrderWorkbenchForm(Forms.FormTableDefinition table) +{ + const double labelX = 24; + const double leftX = 172; + const double rightLabelX = 420; + const double rightX = 556; + const double rowHeight = 34; + const double labelWidth = 132; + const double fieldWidth = 220; + const double top = 24; + const double step = 48; + + var controls = new List(); + void AddField(double y, string fieldName, string label, string controlType, IReadOnlyDictionary? props = null, double x = leftX, double labelPos = labelX, double width = fieldWidth) + { + var mergedProps = props is null + ? new Dictionary() + : new Dictionary(props); + + if (controlType == "checkbox") + mergedProps["text"] = label; + else if (controlType is "text" or "number" or "textarea") + mergedProps.TryAdd("placeholder", label); + + controls.Add(BoundControl(fieldName, controlType, x, y, width, rowHeight, mergedProps, controlType == "computed" ? "OneWay" : "TwoWay")); + } + + AddField(top + (step * 0), "order_number", "Order Number", "text"); + AddField(top + (step * 0), "customer_id", "Customer", "lookup", LookupProps("customers", "name", "id", "Select customer"), rightX, rightLabelX); + AddField(top + (step * 1), "warehouse_id", "Warehouse", "lookup", LookupProps("warehouses", "warehouse_code", "id", "Select warehouse")); + AddField(top + (step * 2), "required_ship_date", "Required Ship", "date"); + AddField(top + (step * 2), "status", "Status", "text", null, rightX, rightLabelX); + AddField(top + (step * 3), "is_expedited", "Expedited", "checkbox"); + AddField(top + (step * 3), "total_amount", "Order Total", "number", NumberProps(min: 0), rightX, rightLabelX); + + controls.Add(BoundControl("notes", "textarea", leftX, top + (step * 4), 756, 72, new Dictionary { ["placeholder"] = "Planner notes and exceptions." })); + + double summaryY = top + (step * 6); + controls.Add(LabelControl("Remaining Units", "remaining_units_total", labelX, summaryY, labelWidth, rowHeight)); + AddField(summaryY, "remaining_units_total", "Remaining Units", "computed", ComputedProps("=SUM(order_lines.ordered_qty) - SUM(order_lines.shipped_qty)", "0"), leftX, labelX, 160); + + IReadOnlyList tabs = + [ + new( + Id: "lines", + Label: "Lines", + ChildTable: "order_lines", + ForeignKeyField: "order_id", + ParentKeyField: "id", + VisibleColumns: ["line_number", "product_id", "ordered_qty", "allocated_qty", "line_total"], + AllowAdd: true, + AllowEdit: true, + AllowDelete: true, + ChildTabs: []) + ]; + + controls.Add(new Forms.ControlDefinition( + ControlId: "order-tabs", + ControlType: "childtabs", + Rect: new Forms.Rect(24, 360, 904, 320), + Binding: null, + Props: new Forms.PropertyBag(new Dictionary + { + ["tabs"] = Forms.ChildTabConfigMapper.ToPropertyBag(tabs), + }), + ValidationOverride: null)); + + return new Forms.FormDefinition( + FormId: "orders-workbench", + Name: "Order Workbench", + TableName: table.TableName, + DefinitionVersion: 1, + SourceSchemaSignature: table.SourceSchemaSignature, + Layout: StandardFormLayout(), + Controls: controls); +} + +static Forms.FormDefinition CreatePurchaseOrderReceivingForm(Forms.FormTableDefinition table) +{ + const double labelX = 24; + const double leftX = 172; + const double rightLabelX = 420; + const double rightX = 556; + const double rowHeight = 34; + const double labelWidth = 132; + const double fieldWidth = 220; + const double top = 24; + const double step = 48; + + var controls = new List(); + void AddField(double y, string fieldName, string label, string controlType, IReadOnlyDictionary? props = null, double x = leftX, double labelPos = labelX, double width = fieldWidth) + { + var mergedProps = props is null + ? new Dictionary() + : new Dictionary(props); + + if (controlType == "checkbox") + mergedProps["text"] = label; + else if (controlType is "text" or "number" or "textarea") + mergedProps.TryAdd("placeholder", label); + + controls.Add(BoundControl(fieldName, controlType, x, y, width, rowHeight, mergedProps, controlType == "computed" ? "OneWay" : "TwoWay")); + } + + AddField(top + (step * 0), "po_number", "PO Number", "text"); + AddField(top + (step * 0), "supplier_id", "Supplier", "lookup", LookupProps("suppliers", "name", "id", "Select supplier"), rightX, rightLabelX); + AddField(top + (step * 1), "warehouse_id", "Warehouse", "lookup", LookupProps("warehouses", "warehouse_code", "id", "Select warehouse")); + AddField(top + (step * 1), "expected_date", "Expected", "date", null, rightX, rightLabelX); + AddField(top + (step * 2), "status", "Status", "text"); + AddField(top + (step * 2), "priority_receiving", "Priority", "checkbox", null, rightX, rightLabelX); + + controls.Add(BoundControl("notes", "textarea", leftX, top + (step * 3), 756, 72, new Dictionary { ["placeholder"] = "Receiving notes and dock exceptions." })); + + double summaryY = top + (step * 5); + controls.Add(LabelControl("Outstanding", "outstanding_units_total", labelX, summaryY, labelWidth, rowHeight)); + AddField(summaryY, "outstanding_units_total", "Outstanding", "computed", ComputedProps("=SUM(purchase_order_lines.ordered_qty) - SUM(purchase_order_lines.received_qty)", "0"), leftX, labelX, 160); + + controls.Add(new Forms.ControlDefinition( + ControlId: "po-lines", + ControlType: "datagrid", + Rect: new Forms.Rect(24, 320, 904, 280), + Binding: null, + Props: new Forms.PropertyBag(new Dictionary + { + ["childTable"] = "purchase_order_lines", + ["foreignKeyField"] = "purchase_order_id", + ["parentKeyField"] = "id", + ["visibleColumns"] = new object?[] { "product_id", "ordered_qty" }, + ["allowAdd"] = true, + ["allowEdit"] = true, + ["allowDelete"] = true, + }), + ValidationOverride: null)); + + return new Forms.FormDefinition( + FormId: "purchase-orders-receiving", + Name: "Purchase Order Receiving", + TableName: table.TableName, + DefinitionVersion: 1, + SourceSchemaSignature: table.SourceSchemaSignature, + Layout: StandardFormLayout(), + Controls: controls); +} + +static Forms.FormDefinition CreateReturnIntakeForm(Forms.FormTableDefinition table) +{ + const double labelX = 24; + const double leftX = 172; + const double rightLabelX = 420; + const double rightX = 556; + const double rowHeight = 34; + const double fieldWidth = 220; + const double top = 24; + const double step = 48; + + var controls = new List(); + void AddField(double y, string fieldName, string label, string controlType, IReadOnlyDictionary? props = null, double x = leftX, double labelPos = labelX, double width = fieldWidth) + { + var mergedProps = props is null + ? new Dictionary() + : new Dictionary(props); + + if (controlType == "checkbox") + mergedProps["text"] = label; + else if (controlType is "text" or "number" or "textarea") + mergedProps.TryAdd("placeholder", label); + + controls.Add(BoundControl(fieldName, controlType, x, y, width, rowHeight, mergedProps)); + } + + AddField(top + (step * 0), "return_number", "Return Number", "text"); + AddField(top + (step * 0), "order_id", "Order", "lookup", LookupProps("orders", "order_number", "id", "Select order"), rightX, rightLabelX); + AddField(top + (step * 1), "product_id", "Product", "lookup", LookupProps("products", "name", "id", "Select product")); + AddField(top + (step * 1), "warehouse_id", "Warehouse", "lookup", LookupProps("warehouses", "warehouse_code", "id", "Select warehouse"), rightX, rightLabelX); + AddField(top + (step * 2), "requested_date", "Requested", "date"); + AddField(top + (step * 2), "received_date", "Received", "date", null, rightX, rightLabelX); + AddField(top + (step * 3), "quantity", "Quantity", "number", NumberProps(min: 0)); + AddField(top + (step * 3), "status", "Status", "text", null, rightX, rightLabelX); + AddField(top + (step * 4), "disposition", "Disposition", "text"); + AddField(top + (step * 4), "requires_qc", "Requires QC", "checkbox", null, rightX, rightLabelX); + + controls.Add(BoundControl("reason", "textarea", leftX, top + (step * 5), 756, 60, new Dictionary { ["placeholder"] = "Return reason." })); + + controls.Add(BoundControl("notes", "textarea", leftX, top + (step * 7), 756, 72, new Dictionary { ["placeholder"] = "Inspection notes." })); + + return new Forms.FormDefinition( + FormId: "returns-intake", + Name: "Return Intake", + TableName: table.TableName, + DefinitionVersion: 1, + SourceSchemaSignature: table.SourceSchemaSignature, + Layout: StandardFormLayout(), + Controls: controls); +} + +static Forms.LayoutDefinition StandardFormLayout() + => new("absolute", 8, true, [new Forms.Breakpoint("md", 0, null)]); + +static Forms.ControlDefinition LabelControl(string text, string forField, double x, double y, double width, double height) + => new( + ControlId: $"{CompactId(forField)}-lbl-{(int)y}", + ControlType: "label", + Rect: new Forms.Rect(x, y, width, height), + Binding: null, + Props: new Forms.PropertyBag(new Dictionary + { + ["text"] = text, + ["forField"] = forField, + }), + ValidationOverride: null); + +static Forms.ControlDefinition BoundControl( + string fieldName, + string controlType, + double x, + double y, + double width, + double height, + IReadOnlyDictionary? props = null, + string mode = "TwoWay") + => new( + ControlId: $"{CompactId(fieldName)}-{CompactId(controlType)}", + ControlType: controlType, + Rect: new Forms.Rect(x, y, width, height), + Binding: new Forms.BindingDefinition(fieldName, mode), + Props: new Forms.PropertyBag(props ?? new Dictionary()), + ValidationOverride: null); + +static string CompactId(string value) + => value.Replace("_", "-", StringComparison.Ordinal).ToLowerInvariant(); + +static IReadOnlyDictionary LookupProps(string tableName, string displayField, string valueField, string placeholder) + => new Dictionary + { + ["lookupTable"] = tableName, + ["displayField"] = displayField, + ["valueField"] = valueField, + ["placeholder"] = placeholder, + }; + +static IReadOnlyDictionary NumberProps(double? min = null, double? max = null) +{ + var props = new Dictionary(); + if (min.HasValue) + props["min"] = min.Value; + if (max.HasValue) + props["max"] = max.Value; + return props; +} + +static IReadOnlyDictionary ComputedProps(string formula, string format) + => new Dictionary + { + ["formula"] = formula, + ["format"] = format, + ["readOnly"] = true, + }; + +static Reports.ReportDefinition CreateShipmentManifestReport(Reports.ReportSourceDefinition source) +{ + List bands = + [ + new( + "report-header", + ReportBandKind.ReportHeader, + 36, + GroupId: null, + Controls: + [ + ReportLabel("report-title", "report-header", 24, 6, 400, 24, "Shipment Manifest", fontSize: 20, fontWeight: "600") + ]), + new( + "page-header", + ReportBandKind.PageHeader, + 22, + GroupId: null, + Controls: + [ + ReportLabel("page-sku", "page-header", 24, 2, 90, 18, "SKU", fontWeight: "600"), + ReportLabel("page-product", "page-header", 126, 2, 220, 18, "Product", fontWeight: "600"), + ReportLabel("page-qty", "page-header", 362, 2, 80, 18, "Qty", fontWeight: "600"), + ReportLabel("page-value", "page-header", 456, 2, 90, 18, "Line Total", fontWeight: "600"), + ReportLabel("page-tracking", "page-header", 566, 2, 180, 18, "Tracking", fontWeight: "600") + ]), + new( + "shipment-group-header", + ReportBandKind.GroupHeader, + 40, + GroupId: "shipment-group", + Controls: + [ + ReportLabel("gh-shipment-label", "shipment-group-header", 24, 2, 80, 18, "Shipment", fontWeight: "600"), + ReportBoundText("gh-shipment-number", "shipment-group-header", 108, 2, 120, 18, "shipment_number"), + ReportLabel("gh-customer-label", "shipment-group-header", 246, 2, 70, 18, "Customer", fontWeight: "600"), + ReportBoundText("gh-customer", "shipment-group-header", 320, 2, 220, 18, "customer_name"), + ReportLabel("gh-carrier-label", "shipment-group-header", 24, 20, 70, 18, "Carrier", fontWeight: "600"), + ReportBoundText("gh-carrier", "shipment-group-header", 108, 20, 120, 18, "carrier_name"), + ReportLabel("gh-order-label", "shipment-group-header", 246, 20, 60, 18, "Order", fontWeight: "600"), + ReportBoundText("gh-order", "shipment-group-header", 320, 20, 120, 18, "order_number"), + ReportLabel("gh-tracking-label", "shipment-group-header", 456, 20, 70, 18, "Tracking", fontWeight: "600"), + ReportBoundText("gh-tracking", "shipment-group-header", 530, 20, 216, 18, "tracking_number") + ]), + new( + "detail", + ReportBandKind.Detail, + 22, + GroupId: null, + Controls: + [ + ReportBoundText("detail-sku", "detail", 24, 2, 90, 18, "sku"), + ReportBoundText("detail-product", "detail", 126, 2, 220, 18, "product_name"), + ReportBoundText("detail-qty", "detail", 362, 2, 80, 18, "quantity_shipped", formatString: "0"), + ReportBoundText("detail-total", "detail", 456, 2, 90, 18, "line_total", formatString: "F2"), + ReportBoundText("detail-tracking", "detail", 566, 2, 180, 18, "tracking_number") + ]), + new( + "shipment-group-footer", + ReportBandKind.GroupFooter, + 24, + GroupId: "shipment-group", + Controls: + [ + ReportLabel("gf-items-label", "shipment-group-footer", 320, 2, 120, 18, "Shipment Units", fontWeight: "600"), + ReportCalculated("gf-items", "shipment-group-footer", 442, 2, 80, 18, "=SUM(quantity_shipped)", "0"), + ReportLabel("gf-value-label", "shipment-group-footer", 540, 2, 90, 18, "Shipment Value", fontWeight: "600"), + ReportCalculated("gf-value", "shipment-group-footer", 632, 2, 100, 18, "=SUM(line_total)", "F2") + ]), + StandardPageFooter() + ]; + + return new Reports.ReportDefinition( + ReportId: "shipment-manifest", + Name: "Shipment Manifest", + Source: new Reports.ReportSourceReference(source.Kind, source.Name), + DefinitionVersion: 1, + SourceSchemaSignature: source.SourceSchemaSignature, + PageSettings: Reports.ReportPageSettings.DefaultLetterPortrait, + Groups: [new Reports.ReportGroupDefinition("shipment-group", "shipment_number")], + Sorts: [new Reports.ReportSortDefinition("shipment_number"), new Reports.ReportSortDefinition("sku")], + Bands: bands); +} + +static Reports.ReportDefinition CreateLowStockReport(Reports.ReportSourceDefinition source) +{ + List bands = + [ + new( + "report-header", + ReportBandKind.ReportHeader, + 36, + GroupId: null, + Controls: + [ + ReportLabel("report-title", "report-header", 24, 6, 320, 24, "Low Stock Watch", fontSize: 20, fontWeight: "600") + ]), + new( + "page-header", + ReportBandKind.PageHeader, + 22, + GroupId: null, + Controls: + [ + ReportLabel("page-sku", "page-header", 24, 2, 80, 18, "SKU", fontWeight: "600"), + ReportLabel("page-product", "page-header", 112, 2, 190, 18, "Product", fontWeight: "600"), + ReportLabel("page-supplier", "page-header", 312, 2, 140, 18, "Supplier", fontWeight: "600"), + ReportLabel("page-available", "page-header", 462, 2, 70, 18, "Avail", fontWeight: "600"), + ReportLabel("page-inbound", "page-header", 540, 2, 70, 18, "Inbound", fontWeight: "600"), + ReportLabel("page-reorder", "page-header", 618, 2, 70, 18, "Reorder", fontWeight: "600"), + ReportLabel("page-shortage", "page-header", 696, 2, 70, 18, "Shortage", fontWeight: "600") + ]), + new( + "warehouse-group-header", + ReportBandKind.GroupHeader, + 24, + GroupId: "warehouse-group", + Controls: + [ + ReportLabel("warehouse-label", "warehouse-group-header", 24, 2, 90, 18, "Warehouse", fontWeight: "600"), + ReportBoundText("warehouse-code", "warehouse-group-header", 118, 2, 100, 18, "warehouse_code"), + ReportBoundText("warehouse-name", "warehouse-group-header", 226, 2, 240, 18, "warehouse_name") + ]), + new( + "detail", + ReportBandKind.Detail, + 22, + GroupId: null, + Controls: + [ + ReportBoundText("detail-sku", "detail", 24, 2, 80, 18, "sku"), + ReportBoundText("detail-product", "detail", 112, 2, 190, 18, "product_name"), + ReportBoundText("detail-supplier", "detail", 312, 2, 140, 18, "supplier_name"), + ReportBoundText("detail-available", "detail", 462, 2, 70, 18, "available_qty", formatString: "0"), + ReportBoundText("detail-inbound", "detail", 540, 2, 70, 18, "inbound_qty", formatString: "0"), + ReportBoundText("detail-reorder", "detail", 618, 2, 70, 18, "reorder_point", formatString: "0"), + ReportBoundText("detail-shortage", "detail", 696, 2, 70, 18, "shortage_qty", formatString: "0") + ]), + new( + "warehouse-group-footer", + ReportBandKind.GroupFooter, + 22, + GroupId: "warehouse-group", + Controls: + [ + ReportLabel("footer-shortage-label", "warehouse-group-footer", 520, 2, 120, 18, "Total Shortage", fontWeight: "600"), + ReportCalculated("footer-shortage", "warehouse-group-footer", 648, 2, 80, 18, "=SUM(shortage_qty)", "0") + ]), + StandardPageFooter() + ]; + + return new Reports.ReportDefinition( + ReportId: "low-stock-watch", + Name: "Low Stock Watch", + Source: new Reports.ReportSourceReference(source.Kind, source.Name), + DefinitionVersion: 1, + SourceSchemaSignature: source.SourceSchemaSignature, + PageSettings: Reports.ReportPageSettings.DefaultLetterPortrait, + Groups: [new Reports.ReportGroupDefinition("warehouse-group", "warehouse_code")], + Sorts: [new Reports.ReportSortDefinition("warehouse_code"), new Reports.ReportSortDefinition("shortage_qty", Descending: true), new Reports.ReportSortDefinition("sku")], + Bands: bands); +} + +static Reports.ReportDefinition CreateOpenOrderQueueReport(Reports.ReportSourceDefinition source) +{ + List bands = + [ + new( + "report-header", + ReportBandKind.ReportHeader, + 36, + GroupId: null, + Controls: + [ + ReportLabel("report-title", "report-header", 24, 6, 360, 24, "Open Order Queue", fontSize: 20, fontWeight: "600") + ]), + new( + "page-header", + ReportBandKind.PageHeader, + 22, + GroupId: null, + Controls: + [ + ReportLabel("page-order", "page-header", 24, 2, 90, 18, "Order", fontWeight: "600"), + ReportLabel("page-customer", "page-header", 126, 2, 180, 18, "Customer", fontWeight: "600"), + ReportLabel("page-warehouse", "page-header", 318, 2, 90, 18, "Warehouse", fontWeight: "600"), + ReportLabel("page-status", "page-header", 420, 2, 80, 18, "Status", fontWeight: "600"), + ReportLabel("page-due", "page-header", 512, 2, 100, 18, "Required", fontWeight: "600"), + ReportLabel("page-value", "page-header", 624, 2, 90, 18, "Value", fontWeight: "600") + ]), + new( + "detail", + ReportBandKind.Detail, + 22, + GroupId: null, + Controls: + [ + ReportBoundText("detail-order", "detail", 24, 2, 90, 18, "order_number"), + ReportBoundText("detail-customer", "detail", 126, 2, 180, 18, "customer_name"), + ReportBoundText("detail-warehouse", "detail", 318, 2, 90, 18, "warehouse_code"), + ReportBoundText("detail-status", "detail", 420, 2, 80, 18, "order_status"), + ReportBoundText("detail-required", "detail", 512, 2, 100, 18, "required_ship_date"), + ReportBoundText("detail-value", "detail", 624, 2, 90, 18, "total_amount", formatString: "F2") + ]), + new( + "report-footer", + ReportBandKind.ReportFooter, + 24, + GroupId: null, + Controls: + [ + ReportLabel("footer-value-label", "report-footer", 566, 2, 120, 18, "Open Order Value", fontWeight: "600"), + ReportCalculated("footer-value", "report-footer", 694, 2, 90, 18, "=SUM(total_amount)", "F2") + ]), + StandardPageFooter() + ]; + + return new Reports.ReportDefinition( + ReportId: "open-order-queue", + Name: "Open Order Queue", + Source: new Reports.ReportSourceReference(source.Kind, source.Name), + DefinitionVersion: 1, + SourceSchemaSignature: source.SourceSchemaSignature, + PageSettings: Reports.ReportPageSettings.DefaultLetterPortrait, + Groups: [], + Sorts: [new Reports.ReportSortDefinition("required_ship_date"), new Reports.ReportSortDefinition("priority_code", Descending: true)], + Bands: bands); +} + +static Reports.ReportBandDefinition StandardPageFooter() + => new( + "page-footer", + ReportBandKind.PageFooter, + 22, + GroupId: null, + Controls: + [ + ReportCalculated("footer-date", "page-footer", 24, 2, 180, 18, "=PrintDate", "g"), + ReportCalculated("footer-page", "page-footer", 690, 2, 90, 18, "=PageNumber", null, "Page ", textAlign: "right") + ]); + +static Reports.ReportControlDefinition ReportLabel( + string id, + string bandId, + double x, + double y, + double width, + double height, + string text, + long? fontSize = null, + string? fontWeight = null) +{ + var props = new Dictionary { ["text"] = text }; + if (fontSize.HasValue) + props["fontSize"] = fontSize.Value; + if (!string.IsNullOrWhiteSpace(fontWeight)) + props["fontWeight"] = fontWeight; + + return new Reports.ReportControlDefinition( + id, + ReportControlType.Label, + bandId, + new Reports.Rect(x, y, width, height), + BoundFieldName: null, + Expression: null, + FormatString: null, + Props: new Reports.PropertyBag(props)); +} + +static Reports.ReportControlDefinition ReportBoundText( + string id, + string bandId, + double x, + double y, + double width, + double height, + string fieldName, + string? formatString = null) + => new( + id, + ReportControlType.BoundText, + bandId, + new Reports.Rect(x, y, width, height), + BoundFieldName: fieldName, + Expression: null, + FormatString: formatString, + Props: Reports.PropertyBag.Empty); + +static Reports.ReportControlDefinition ReportCalculated( + string id, + string bandId, + double x, + double y, + double width, + double height, + string expression, + string? formatString, + string? prefix = null, + string? textAlign = null) +{ + var props = new Dictionary(); + if (!string.IsNullOrWhiteSpace(prefix)) + props["prefix"] = prefix; + if (!string.IsNullOrWhiteSpace(textAlign)) + props["textAlign"] = textAlign; + + return new Reports.ReportControlDefinition( + id, + ReportControlType.CalculatedText, + bandId, + new Reports.Rect(x, y, width, height), + BoundFieldName: null, + Expression: expression, + FormatString: formatString, + Props: new Reports.PropertyBag(props)); +} + +file sealed record StoredProcedureFile( + string Name, + string BodySql, + IReadOnlyList Parameters, + string? Description, + bool IsEnabled); + +file sealed record StoredProcedureParameterFile( + string Name, + string Type, + bool Required, + JsonElement? Default, + string? Description); + +file sealed record SavedQueryFile( + string Name, + string SqlText); + +file sealed record ScannerSessionDocument( + string DeviceId, + string WarehouseCode, + string OperatorName, + string SessionStatus, + ScannerWaveState CurrentWave, + string[] ScannedSkus, + string[] Tags, + string LastActivityUtc); + +file sealed record ScannerWaveState( + string OrderNumber, + string Phase, + string StartedUtc); + +file sealed record WebhookArchiveDocument( + string Provider, + string EventType, + string ReceivedUtc, + WebhookHeaders Headers, + string[] Tags, + string PayloadJson); + +file sealed record WebhookHeaders( + string OrderNumber, + string? ShipmentNumber); + +file sealed record FullTextSeedSummary( + IReadOnlyList Hits); + +file sealed record CollectionSeedSummary( + int ScannerSessionCount, + int WebhookCount, + IReadOnlyList ScannerSessionMatches, + IReadOnlyList WebhookTagMatches); + +file sealed record PipelineSeedSummary( + IReadOnlyList RunSummaries, + string ExportPath); diff --git a/samples/fulfillment-hub/README.md b/samples/fulfillment-hub/README.md new file mode 100644 index 00000000..ce57e077 --- /dev/null +++ b/samples/fulfillment-hub/README.md @@ -0,0 +1,295 @@ +# Fulfillment Hub + +Fulfillment Hub is a guided operations sample for CSharpDB. Instead of showing isolated features, it gives you one warehouse story and lets the database follow that story from inbound receiving to outbound shipment to returns, with forms, reports, pipelines, collections, saved queries, procedures, triggers, and full-text search all attached to the same working set. + +If you want one sample that teaches the platform as a whole, start here. + +## Start Here + +Run the seeder: + +```bash +dotnet run --project samples/fulfillment-hub/FulfillmentHubSample.csproj +``` + +Each run creates a fresh demo database here: + +```text +samples/fulfillment-hub/bin/Debug/net10.0/fulfillment-hub-demo.db +``` + +The seeder does more than load SQL. It also: + +- creates the relational schema and fixed snapshot data +- imports stored procedures and saved queries +- stores Admin forms and reports +- stores and runs pipeline packages +- seeds typed collections +- builds a full-text index over operational playbooks + +When the run finishes, keep that database and explore it in the Admin UI, the CLI, or through `CSharpDB.Client`. + +## How To Use This Sample + +Use the sample in this order: + +1. Run the seeder. +2. Open the generated database in the Admin project or query it through the CLI. +3. Follow the walkthrough below in sequence. +4. After each step, inspect the related table, view, saved query, form, report, or collection. +5. Change a few rows and rerun the same step so you can see which parts of the platform are relational, which parts are metadata-driven, and which parts are procedural. + +This sample is easiest to learn if you treat it like a live operations system, not a pile of setup files. + +## The Story + +It is Friday morning. Seattle and Denver are trying to clear outbound demand before the weekend, Atlanta is handling returns, and a few SKUs are already running tight. Your job is to operate the day using the features seeded by this sample. + +### 1. Start With The Live Queue + +First, look at the order board. This is the operational heartbeat of the sample. + +```sql +SELECT order_number, customer_name, warehouse_code, order_status, required_ship_date +FROM order_fulfillment_board +WHERE order_status IN ('released', 'allocated', 'picking') +ORDER BY required_ship_date, priority_code DESC, order_number; +``` + +What to learn here: + +- `order_fulfillment_board` is a view used as a planner-facing queue +- the data model is relational, but the workflow is operational +- this same queue also feeds the saved query and one report source + +Then compare it to the saved query: + +```sql +EXEC RefreshOperationalStats; +``` + +That procedure returns the current table stats, shortage watch, and open order board in one call. It is the simplest way to see how stored procedures can package operational read models for Admin users. + +### 2. Find The Inventory Problem Before You Allocate Anything + +Now switch to the shortage watch. + +```sql +SELECT warehouse_code, sku, product_name, available_qty, inbound_qty, reorder_point, shortage_qty +FROM low_stock_watch +WHERE shortage_qty > 0 +ORDER BY shortage_qty DESC, warehouse_code, sku; +``` + +This is where the sample starts teaching you to combine features: + +- `low_stock_watch` is a view +- it is also a saved query target +- it is also a report source +- it is also exported by a stored pipeline package + +If you are in the Admin UI, open the `Low Stock Watch` report after querying this view. You should see the same shortage story represented as a print-style artifact instead of a SQL result set. + +### 3. Pull In New Supply Through Pipelines + +The warehouse receives new inbound files from external systems. The sample already stores and runs these pipeline packages for you: + +- `supplier-receipts-import` +- `marketplace-orders-import` +- `low-stock-export` + +Inspect the staged results: + +```sql +SELECT * +FROM supplier_receipts_stage +ORDER BY id; + +SELECT * +FROM marketplace_orders_stage +ORDER BY id; +``` + +What to learn here: + +- pipelines are not separate from the database story; they feed the same operational model +- one package imports CSV, one imports JSON, and one exports a query result to CSV +- the run history is stored, so pipeline execution becomes part of the system record + +Check the generated export file too: + +```text +samples/fulfillment-hub/bin/Debug/net10.0/generated-output/low-stock-watch.csv +``` + +### 4. Receive A Purchase Order + +Now imagine Seattle receives the replenishment shipment for `PO-9001`. + +Run: + +```sql +EXEC ReceivePurchaseOrder purchaseOrderId=9001; +``` + +Then inspect: + +```sql +SELECT * +FROM purchase_order_receiving_board +WHERE purchase_order_id = 9001 +ORDER BY sku; +``` + +What to learn here: + +- procedures can coordinate multi-statement updates +- one procedure can update base tables, write an event row, and return useful follow-up result sets +- the `Purchase Order Receiving` form is bound to the same purchase-order model and child lines + +In the Admin UI, open the `Purchase Order Receiving` form. This is where the sample becomes more than SQL: you can see lookups, computed totals, and a child data grid hanging off the same underlying schema. + +### 5. Allocate A Waiting Order + +Now that inbound supply is available, allocate a waiting order: + +```sql +EXEC AllocateOrder orderId=7005; +``` + +Then inspect the order again: + +```sql +SELECT * +FROM order_fulfillment_board +WHERE order_id = 7005; +``` + +This step teaches a few things at once: + +- procedures can reserve inventory by updating multiple related tables +- the order workflow is stateful, but the state is still inspectable with plain SQL +- the `Order Workbench` form is the human-facing surface for the same workflow + +Open `orders-workbench` in the Admin UI and look at the order line child tab. That is the same order you just changed procedurally, now visible through form metadata rather than hand-written UI code. + +### 6. Create The Shipment + +Once an order is allocated, ship it: + +```sql +EXEC CreateShipment orderId=7001, shipmentId=8101, shipmentNumber='SHP-8101', carrierId=2; +``` + +Then inspect the shipment report source: + +```sql +SELECT * +FROM shipment_manifest_report_source +WHERE shipment_number = 'SHP-8101' +ORDER BY sku; +``` + +Now open the `Shipment Manifest` report in Admin. + +What to learn here: + +- a report source can be a view tailored for printing +- reports do not need a separate reporting database +- the same operational transaction that creates shipment rows also produces report-ready output + +### 7. Process A Return + +The day is not only outbound. Atlanta receives returns too. + +Run: + +```sql +EXEC RecordReturn returnId=8502, newStatus='closed', disposition='restock'; +``` + +Then inspect: + +```sql +SELECT * +FROM return_queue +ORDER BY requested_date DESC, return_number; +``` + +This is where the sample shows that reverse logistics is not a separate subsystem. Returns live in the same operational model, can update stock, and can be managed with the `Return Intake` form. + +### 8. Check The Operational Audit Trail + +Several actions in the sample write to `ops_events` through triggers and procedures. + +Inspect that stream: + +```sql +SELECT entity_type, entity_id, event_type, event_date, actor_name, details +FROM ops_events +ORDER BY id DESC +LIMIT 20; +``` + +What to learn here: + +- triggers are used for automatic audit-style events +- procedures can add richer business events with actor and narrative context +- the event log becomes a simple operational history that can be queried without extra infrastructure + +### 9. Use Full-Text Search To Find The Playbook + +At some point, the operator needs guidance, not just data. That is why the sample also seeds `ops_playbooks` and creates a full-text index. + +Search for the receiving issue through the engine API: + +```csharp +await using var db = await Database.OpenAsync("samples/fulfillment-hub/bin/Debug/net10.0/fulfillment-hub-demo.db"); +var hits = await db.SearchAsync("fts_ops_playbooks", "partial receipt"); +``` + +This part is important. The sample is not just showing full-text search in isolation. It is showing how documentation, runbooks, and live operations can sit in the same database. + +### 10. Look At The Collections Side + +Not every useful artifact belongs in a rigid relational table. The sample also seeds two collections: + +- `scanner_sessions` +- `webhook_archive` + +These are there to show where typed or semi-structured operational data fits. + +The sample seeder verifies them with collection path lookups such as: + +- `CurrentWave.OrderNumber` +- `$.tags[]` + +What to learn here: + +- relational tables still own the transactional model +- collections work well for transient device state, webhook payloads, and nested documents +- indexed path queries let those documents stay queryable without forcing them into awkward relational tables + +## How The Files Map To The Story + +- [schema.sql](C:/Users/maxim/source/Code/CSharpDB/samples/fulfillment-hub/schema.sql) creates the relational model, views, triggers, and base snapshot +- [procedures.json](C:/Users/maxim/source/Code/CSharpDB/samples/fulfillment-hub/procedures.json) defines the main operational actions +- [saved-queries.json](C:/Users/maxim/source/Code/CSharpDB/samples/fulfillment-hub/saved-queries.json) stores the queue-style queries that make sense in Admin +- [queries.sql](C:/Users/maxim/source/Code/CSharpDB/samples/fulfillment-hub/queries.sql) is the workbook for exploration and learning +- [pipelines](C:/Users/maxim/source/Code/CSharpDB/samples/fulfillment-hub/pipelines/low-stock-export.json) contains stored pipeline definitions +- [imports](C:/Users/maxim/source/Code/CSharpDB/samples/fulfillment-hub/imports/supplier-receipts.csv) contains source files for those pipelines +- [Program.cs](C:/Users/maxim/source/Code/CSharpDB/samples/fulfillment-hub/Program.cs) seeds everything that is not pure DDL: forms, reports, pipeline packages, collections, and full-text indexes + +## A Good Learning Loop + +If you want to really learn the platform instead of just running the sample once, use this loop: + +1. Run the sample. +2. Read one procedure in `procedures.json`. +3. Execute it. +4. Inspect the changed tables and views. +5. Open the related form or report in Admin. +6. Look at the same behavior in `Program.cs` to see how metadata was seeded. +7. Change the schema or procedure and rerun the sample. + +That loop will teach you more about CSharpDB than reading the feature list in isolation. diff --git a/samples/fulfillment-hub/imports/marketplace-orders.json b/samples/fulfillment-hub/imports/marketplace-orders.json new file mode 100644 index 00000000..66931843 --- /dev/null +++ b/samples/fulfillment-hub/imports/marketplace-orders.json @@ -0,0 +1,50 @@ +[ + { + "externalOrderId": "MKP-10001", + "customerEmail": "buyers@driftcoffee.example", + "customerName": "Drift Coffee", + "warehouseCode": "SEA-FC", + "orderDate": "2026-04-24", + "sku": "BOT-220", + "quantity": "4", + "unitPrice": "24.00", + "salesChannel": "marketplace", + "status": "ready" + }, + { + "externalOrderId": "MKP-10001", + "customerEmail": "buyers@driftcoffee.example", + "customerName": "Drift Coffee", + "warehouseCode": "SEA-FC", + "orderDate": "2026-04-24", + "sku": "BOT-220", + "quantity": "4", + "unitPrice": "24.00", + "salesChannel": "marketplace", + "status": "ready" + }, + { + "externalOrderId": "MKP-10002", + "customerEmail": "ops@alpineoutdoor.example", + "customerName": "Alpine Outdoor", + "warehouseCode": "SEA-FC", + "orderDate": "2026-04-24", + "sku": "TRJ-100", + "quantity": "2", + "unitPrice": "89.00", + "salesChannel": "marketplace", + "status": "ready" + }, + { + "externalOrderId": "MKP-10003", + "customerEmail": "warehouse@cedarrobotics.example", + "customerName": "Cedar Robotics", + "warehouseCode": "ATL-RTN", + "orderDate": "2026-04-24", + "sku": "CCS-810", + "quantity": "1", + "unitPrice": "48.30", + "salesChannel": "marketplace", + "status": "hold" + } +] diff --git a/samples/fulfillment-hub/pipelines/low-stock-export.json b/samples/fulfillment-hub/pipelines/low-stock-export.json new file mode 100644 index 00000000..078bc540 --- /dev/null +++ b/samples/fulfillment-hub/pipelines/low-stock-export.json @@ -0,0 +1,35 @@ +{ + "name": "low-stock-export", + "version": "1.0.0", + "description": "Exports the live low-stock watchlist to a CSV file for morning warehouse reviews.", + "source": { + "kind": "sqlQuery", + "queryText": "SELECT warehouse_code, sku, product_name, supplier_name, available_qty, inbound_qty, reorder_point, shortage_qty FROM low_stock_watch WHERE shortage_qty > 0 ORDER BY warehouse_code, shortage_qty DESC, sku;" + }, + "transforms": [ + { + "kind": "select", + "selectColumns": [ + "warehouse_code", + "sku", + "product_name", + "supplier_name", + "available_qty", + "inbound_qty", + "reorder_point", + "shortage_qty" + ] + } + ], + "destination": { + "kind": "csvFile", + "path": "__OUTPUT_DIR__/low-stock-watch.csv", + "overwrite": true + }, + "options": { + "batchSize": 100, + "errorMode": "failFast", + "checkpointInterval": 50, + "maxRejects": 0 + } +} diff --git a/samples/fulfillment-hub/pipelines/marketplace-orders-import.json b/samples/fulfillment-hub/pipelines/marketplace-orders-import.json new file mode 100644 index 00000000..b26ccdf6 --- /dev/null +++ b/samples/fulfillment-hub/pipelines/marketplace-orders-import.json @@ -0,0 +1,92 @@ +{ + "name": "marketplace-orders-import", + "version": "1.0.0", + "description": "Imports marketplace order payloads into a staging table for review and conversion.", + "source": { + "kind": "jsonFile", + "path": "__SAMPLE_DIR__/imports/marketplace-orders.json" + }, + "transforms": [ + { + "kind": "rename", + "renameMappings": [ + { + "source": "externalOrderId", + "target": "external_order_id" + }, + { + "source": "customerEmail", + "target": "customer_email" + }, + { + "source": "customerName", + "target": "customer_name" + }, + { + "source": "warehouseCode", + "target": "warehouse_code" + }, + { + "source": "orderDate", + "target": "order_date" + }, + { + "source": "unitPrice", + "target": "unit_price" + }, + { + "source": "salesChannel", + "target": "sales_channel" + }, + { + "source": "status", + "target": "import_status" + } + ] + }, + { + "kind": "cast", + "castMappings": [ + { + "column": "quantity", + "targetType": "integer" + }, + { + "column": "unit_price", + "targetType": "real" + } + ] + }, + { + "kind": "filter", + "filterExpression": "import_status == 'ready'" + }, + { + "kind": "derive", + "derivedColumns": [ + { + "name": "channel_snapshot", + "expression": "sales_channel" + } + ] + }, + { + "kind": "deduplicate", + "deduplicateKeys": [ + "external_order_id", + "sku" + ] + } + ], + "destination": { + "kind": "cSharpDbTable", + "tableName": "marketplace_orders_stage", + "overwrite": true + }, + "options": { + "batchSize": 100, + "errorMode": "skipBadRows", + "checkpointInterval": 50, + "maxRejects": 10 + } +} diff --git a/samples/fulfillment-hub/pipelines/supplier-receipts-import.json b/samples/fulfillment-hub/pipelines/supplier-receipts-import.json new file mode 100644 index 00000000..632d95c8 --- /dev/null +++ b/samples/fulfillment-hub/pipelines/supplier-receipts-import.json @@ -0,0 +1,95 @@ +{ + "name": "supplier-receipts-import", + "version": "1.0.0", + "description": "Imports supplier receipt files into the operational receipt staging table.", + "source": { + "kind": "csvFile", + "path": "__SAMPLE_DIR__/imports/supplier-receipts.csv", + "hasHeaderRow": true + }, + "transforms": [ + { + "kind": "select", + "selectColumns": [ + "batch_id", + "supplier", + "warehouse", + "sku", + "qty", + "cost", + "received_date", + "reference", + "status" + ] + }, + { + "kind": "rename", + "renameMappings": [ + { + "source": "batch_id", + "target": "receipt_batch" + }, + { + "source": "supplier", + "target": "supplier_code" + }, + { + "source": "warehouse", + "target": "warehouse_code" + }, + { + "source": "qty", + "target": "received_qty" + }, + { + "source": "cost", + "target": "unit_cost" + }, + { + "source": "reference", + "target": "reference_number" + }, + { + "source": "status", + "target": "ingest_status" + } + ] + }, + { + "kind": "cast", + "castMappings": [ + { + "column": "received_qty", + "targetType": "integer" + }, + { + "column": "unit_cost", + "targetType": "real" + } + ] + }, + { + "kind": "filter", + "filterExpression": "ingest_status == 'posted'" + }, + { + "kind": "deduplicate", + "deduplicateKeys": [ + "receipt_batch", + "sku", + "reference_number" + ] + } + ], + "destination": { + "kind": "cSharpDbTable", + "tableName": "supplier_receipts_stage", + "overwrite": true + }, + "options": { + "batchSize": 100, + "errorMode": "skipBadRows", + "checkpointInterval": 50, + "maxRejects": 10 + } +} diff --git a/samples/fulfillment-hub/procedures.json b/samples/fulfillment-hub/procedures.json new file mode 100644 index 00000000..d4221f70 --- /dev/null +++ b/samples/fulfillment-hub/procedures.json @@ -0,0 +1,195 @@ +[ + { + "name": "AllocateOrder", + "bodySql": "UPDATE inventory_positions\nSET allocated_qty = allocated_qty + (\n SELECT SUM(ol.ordered_qty - ol.allocated_qty)\n FROM order_lines ol\n INNER JOIN orders o ON o.id = ol.order_id\n WHERE ol.order_id = @orderId\n AND o.warehouse_id = inventory_positions.warehouse_id\n AND ol.product_id = inventory_positions.product_id\n AND ol.ordered_qty > ol.allocated_qty\n)\nWHERE EXISTS (\n SELECT 1\n FROM order_lines ol\n INNER JOIN orders o ON o.id = ol.order_id\n WHERE ol.order_id = @orderId\n AND o.warehouse_id = inventory_positions.warehouse_id\n AND ol.product_id = inventory_positions.product_id\n AND ol.ordered_qty > ol.allocated_qty\n);\nUPDATE order_lines\nSET allocated_qty = ordered_qty\nWHERE order_id = @orderId;\nUPDATE orders\nSET status = 'allocated',\n notes = @note\nWHERE id = @orderId;\nINSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details)\nVALUES ('order', @orderId, 'allocated', @allocatedDate, @allocatedBy, @note);\nSELECT *\nFROM order_fulfillment_board\nWHERE order_id = @orderId;\nSELECT sku, product_name, available_qty, shortage_qty\nFROM low_stock_watch\nWHERE warehouse_id = (\n SELECT warehouse_id\n FROM orders\n WHERE id = @orderId\n)\nORDER BY shortage_qty DESC, sku;", + "parameters": [ + { + "name": "orderId", + "type": "INTEGER", + "required": true, + "description": "Sales order identifier." + }, + { + "name": "allocatedDate", + "type": "TEXT", + "required": false, + "default": "2026-04-24", + "description": "Allocation date (YYYY-MM-DD)." + }, + { + "name": "allocatedBy", + "type": "TEXT", + "required": false, + "default": "Wave Planner", + "description": "Planner or automation actor." + }, + { + "name": "note", + "type": "TEXT", + "required": false, + "default": "Allocation released from Fulfillment Hub sample procedure.", + "description": "Operational note written back to the order." + } + ], + "description": "Allocates all unallocated order lines for one sales order, updates inventory reservations, and returns the refreshed order plus low-stock watch rows for the warehouse.", + "isEnabled": true + }, + { + "name": "ReceivePurchaseOrder", + "bodySql": "UPDATE inventory_positions\nSET on_hand_qty = on_hand_qty + (\n SELECT SUM(pol.ordered_qty - pol.received_qty)\n FROM purchase_order_lines pol\n INNER JOIN purchase_orders po ON po.id = pol.purchase_order_id\n WHERE pol.purchase_order_id = @purchaseOrderId\n AND po.warehouse_id = inventory_positions.warehouse_id\n AND pol.product_id = inventory_positions.product_id\n ),\n inbound_qty = inbound_qty - (\n SELECT SUM(pol.ordered_qty - pol.received_qty)\n FROM purchase_order_lines pol\n INNER JOIN purchase_orders po ON po.id = pol.purchase_order_id\n WHERE pol.purchase_order_id = @purchaseOrderId\n AND po.warehouse_id = inventory_positions.warehouse_id\n AND pol.product_id = inventory_positions.product_id\n )\nWHERE EXISTS (\n SELECT 1\n FROM purchase_order_lines pol\n INNER JOIN purchase_orders po ON po.id = pol.purchase_order_id\n WHERE pol.purchase_order_id = @purchaseOrderId\n AND po.warehouse_id = inventory_positions.warehouse_id\n AND pol.product_id = inventory_positions.product_id\n AND pol.ordered_qty > pol.received_qty\n);\nUPDATE purchase_order_lines\nSET received_qty = ordered_qty\nWHERE purchase_order_id = @purchaseOrderId;\nUPDATE purchase_orders\nSET status = 'received',\n notes = @note\nWHERE id = @purchaseOrderId;\nINSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details)\nVALUES ('purchase_order', @purchaseOrderId, 'received', @receivedDate, @receivedBy, @note);\nSELECT *\nFROM purchase_order_receiving_board\nWHERE purchase_order_id = @purchaseOrderId\nORDER BY sku;\nSELECT *\nFROM low_stock_watch\nWHERE warehouse_id = (\n SELECT warehouse_id\n FROM purchase_orders\n WHERE id = @purchaseOrderId\n)\nORDER BY shortage_qty DESC, sku;", + "parameters": [ + { + "name": "purchaseOrderId", + "type": "INTEGER", + "required": true, + "description": "Purchase order identifier." + }, + { + "name": "receivedDate", + "type": "TEXT", + "required": false, + "default": "2026-04-24", + "description": "Receipt date (YYYY-MM-DD)." + }, + { + "name": "receivedBy", + "type": "TEXT", + "required": false, + "default": "Receiving Desk", + "description": "Receiving actor recorded in ops events." + }, + { + "name": "note", + "type": "TEXT", + "required": false, + "default": "Purchase order fully received from sample procedure.", + "description": "Operational note written back to the purchase order." + } + ], + "description": "Receives all outstanding quantities for a purchase order, moves inbound stock into on-hand inventory, and returns the refreshed receiving board plus warehouse shortage rows.", + "isEnabled": true + }, + { + "name": "CreateShipment", + "bodySql": "INSERT INTO shipments (\n id,\n shipment_number,\n order_id,\n carrier_id,\n warehouse_id,\n shipped_date,\n status,\n tracking_number,\n picked_by,\n packed_by\n)\nSELECT\n @shipmentId,\n @shipmentNumber,\n o.id,\n @carrierId,\n o.warehouse_id,\n @shippedDate,\n 'shipped',\n @trackingNumber,\n @pickedBy,\n @packedBy\nFROM orders o\nWHERE o.id = @orderId;\nINSERT INTO shipment_lines (shipment_id, order_line_id, product_id, quantity_shipped, line_total)\nSELECT\n @shipmentId,\n ol.id,\n ol.product_id,\n ol.allocated_qty - ol.shipped_qty,\n (ol.allocated_qty - ol.shipped_qty) * ol.unit_price\nFROM order_lines ol\nWHERE ol.order_id = @orderId\n AND ol.allocated_qty > ol.shipped_qty;\nUPDATE inventory_positions\nSET on_hand_qty = on_hand_qty - (\n SELECT SUM(ol.allocated_qty - ol.shipped_qty)\n FROM order_lines ol\n INNER JOIN orders o ON o.id = ol.order_id\n WHERE ol.order_id = @orderId\n AND o.warehouse_id = inventory_positions.warehouse_id\n AND ol.product_id = inventory_positions.product_id\n ),\n allocated_qty = allocated_qty - (\n SELECT SUM(ol.allocated_qty - ol.shipped_qty)\n FROM order_lines ol\n INNER JOIN orders o ON o.id = ol.order_id\n WHERE ol.order_id = @orderId\n AND o.warehouse_id = inventory_positions.warehouse_id\n AND ol.product_id = inventory_positions.product_id\n )\nWHERE EXISTS (\n SELECT 1\n FROM order_lines ol\n INNER JOIN orders o ON o.id = ol.order_id\n WHERE ol.order_id = @orderId\n AND o.warehouse_id = inventory_positions.warehouse_id\n AND ol.product_id = inventory_positions.product_id\n AND ol.allocated_qty > ol.shipped_qty\n);\nUPDATE order_lines\nSET shipped_qty = allocated_qty\nWHERE order_id = @orderId;\nUPDATE orders\nSET status = 'shipped',\n notes = @note\nWHERE id = @orderId;\nSELECT *\nFROM shipment_manifest_report_source\nWHERE shipment_id = @shipmentId\nORDER BY sku;\nSELECT *\nFROM order_fulfillment_board\nWHERE order_id = @orderId;", + "parameters": [ + { + "name": "orderId", + "type": "INTEGER", + "required": true, + "description": "Sales order identifier." + }, + { + "name": "shipmentId", + "type": "INTEGER", + "required": true, + "description": "Shipment identifier to create." + }, + { + "name": "shipmentNumber", + "type": "TEXT", + "required": true, + "description": "Shipment number." + }, + { + "name": "carrierId", + "type": "INTEGER", + "required": true, + "description": "Carrier identifier." + }, + { + "name": "shippedDate", + "type": "TEXT", + "required": false, + "default": "2026-04-24", + "description": "Ship date (YYYY-MM-DD)." + }, + { + "name": "trackingNumber", + "type": "TEXT", + "required": false, + "default": "TRACK-SAMPLE-001", + "description": "Carrier tracking number." + }, + { + "name": "pickedBy", + "type": "TEXT", + "required": false, + "default": "Wave Picker", + "description": "Picker name." + }, + { + "name": "packedBy", + "type": "TEXT", + "required": false, + "default": "Pack Station 4", + "description": "Packer name." + }, + { + "name": "note", + "type": "TEXT", + "required": false, + "default": "Shipment created from Fulfillment Hub sample procedure.", + "description": "Operational note written back to the sales order." + } + ], + "description": "Creates a shipment for one order, copies all remaining allocated lines into shipment lines, consumes reserved inventory, and returns the manifest rows plus refreshed order status.", + "isEnabled": true + }, + { + "name": "RecordReturn", + "bodySql": "UPDATE returns\nSET status = @newStatus,\n disposition = @disposition,\n received_date = @processedDate,\n notes = @note\nWHERE id = @returnId;\nUPDATE inventory_positions\nSET on_hand_qty = on_hand_qty + (\n SELECT quantity\n FROM returns r\n WHERE r.id = @returnId\n AND r.warehouse_id = inventory_positions.warehouse_id\n AND r.product_id = inventory_positions.product_id\n AND @disposition = 'restock'\n)\nWHERE EXISTS (\n SELECT 1\n FROM returns r\n WHERE r.id = @returnId\n AND r.warehouse_id = inventory_positions.warehouse_id\n AND r.product_id = inventory_positions.product_id\n AND @disposition = 'restock'\n);\nINSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details)\nVALUES ('return', @returnId, 'processed', @processedDate, @processedBy, @note);\nSELECT *\nFROM return_queue\nWHERE return_id = @returnId;\nSELECT *\nFROM low_stock_watch\nWHERE product_id = (\n SELECT product_id\n FROM returns\n WHERE id = @returnId\n)\nORDER BY warehouse_code;", + "parameters": [ + { + "name": "returnId", + "type": "INTEGER", + "required": true, + "description": "Return identifier." + }, + { + "name": "newStatus", + "type": "TEXT", + "required": false, + "default": "closed", + "description": "Updated return status." + }, + { + "name": "disposition", + "type": "TEXT", + "required": false, + "default": "restock", + "description": "Disposition such as restock, inspect, or scrap." + }, + { + "name": "processedDate", + "type": "TEXT", + "required": false, + "default": "2026-04-24", + "description": "Processing date (YYYY-MM-DD)." + }, + { + "name": "processedBy", + "type": "TEXT", + "required": false, + "default": "Returns Desk", + "description": "Return processor." + }, + { + "name": "note", + "type": "TEXT", + "required": false, + "default": "Return processed from Fulfillment Hub sample procedure.", + "description": "Operational note written back to the return." + } + ], + "description": "Processes one return, optionally restocks inventory when the disposition is restock, and returns the refreshed return queue row plus product shortage rows.", + "isEnabled": true + }, + { + "name": "RefreshOperationalStats", + "bodySql": "SELECT table_name, row_count, row_count_is_exact, has_stale_columns\nFROM sys.table_stats\nORDER BY row_count DESC, table_name;\nSELECT warehouse_code, sku, product_name, available_qty, inbound_qty, reorder_point, shortage_qty\nFROM low_stock_watch\nWHERE shortage_qty > 0\nORDER BY shortage_qty DESC, warehouse_code, sku;\nSELECT order_number, customer_name, warehouse_code, order_status, priority_code, total_amount\nFROM order_fulfillment_board\nORDER BY required_ship_date, priority_code DESC, order_number;", + "parameters": [], + "description": "Returns table stats, shortage watch rows, and the live order fulfillment board for planner review.", + "isEnabled": true + } +] diff --git a/samples/fulfillment-hub/queries.sql b/samples/fulfillment-hub/queries.sql new file mode 100644 index 00000000..bb843349 --- /dev/null +++ b/samples/fulfillment-hub/queries.sql @@ -0,0 +1,97 @@ +-- Fulfillment Hub workbook +-- Read-only exploration queries for the seeded operational snapshot. + +SELECT + order_number, + customer_name, + warehouse_code, + order_status, + priority_code, + total_amount +FROM order_fulfillment_board +ORDER BY required_ship_date, priority_code DESC, order_number; + +SELECT + warehouse_code, + sku, + product_name, + available_qty, + inbound_qty, + reorder_point, + shortage_qty +FROM low_stock_watch +WHERE shortage_qty > 0 +ORDER BY shortage_qty DESC, warehouse_code, sku; + +SELECT + po_number, + supplier_name, + warehouse_code, + expected_date, + po_status, + sku, + ordered_qty, + received_qty, + outstanding_qty +FROM purchase_order_receiving_board +WHERE outstanding_qty > 0 +ORDER BY expected_date, po_number, sku; + +SELECT + shipment_number, + shipment_status, + shipped_date, + carrier_name, + order_number, + customer_name, + sku, + quantity_shipped +FROM shipment_manifest_report_source +ORDER BY shipment_number, sku; + +SELECT + return_number, + return_status, + requested_date, + customer_name, + sku, + product_name, + warehouse_code, + quantity, + reason +FROM return_queue +ORDER BY requested_date DESC, return_number; + +SELECT + entity_type, + entity_id, + event_type, + event_date, + actor_name, + details +FROM ops_events +ORDER BY id DESC +LIMIT 20; + +SELECT + table_name, + row_count, + row_count_is_exact, + has_stale_columns +FROM sys.table_stats +ORDER BY row_count DESC, table_name; + +SELECT + object_name, + object_type +FROM sys.objects +ORDER BY object_type, object_name; + +-- Pipeline-run target tables: +SELECT * +FROM supplier_receipts_stage +ORDER BY id; + +SELECT * +FROM marketplace_orders_stage +ORDER BY id; diff --git a/samples/fulfillment-hub/saved-queries.json b/samples/fulfillment-hub/saved-queries.json new file mode 100644 index 00000000..14f11310 --- /dev/null +++ b/samples/fulfillment-hub/saved-queries.json @@ -0,0 +1,22 @@ +[ + { + "name": "Open Order Queue", + "sqlText": "SELECT order_id, order_number, customer_name, warehouse_code, order_date, required_ship_date, order_status, priority_code, is_expedited, total_amount\nFROM order_fulfillment_board\nWHERE order_status IN ('released', 'allocated', 'picking')\nORDER BY is_expedited DESC, required_ship_date, priority_code DESC, order_number;" + }, + { + "name": "Low Stock By Warehouse", + "sqlText": "SELECT warehouse_code, warehouse_name, sku, product_name, supplier_name, available_qty, inbound_qty, reorder_point, shortage_qty\nFROM low_stock_watch\nWHERE shortage_qty > 0\nORDER BY warehouse_code, shortage_qty DESC, sku;" + }, + { + "name": "Returns Awaiting Inspection", + "sqlText": "SELECT return_id, return_number, return_status, requested_date, received_date, customer_name, sku, product_name, warehouse_code, quantity, reason, disposition\nFROM return_queue\nWHERE return_status IN ('requested', 'received')\nORDER BY requested_date DESC, return_number;" + }, + { + "name": "Purchase Orders Due This Week", + "sqlText": "SELECT purchase_order_id, po_number, supplier_name, warehouse_code, expected_date, po_status, priority_receiving, sku, ordered_qty, received_qty, outstanding_qty\nFROM purchase_order_receiving_board\nWHERE po_status IN ('open', 'partial')\n AND expected_date <= '2026-04-30'\nORDER BY expected_date, priority_receiving DESC, po_number, sku;" + }, + { + "name": "Shipment Exceptions", + "sqlText": "SELECT o.order_number, c.name AS customer_name, w.warehouse_code, o.required_ship_date, o.status AS order_status, 0 AS shipment_count, o.notes\nFROM orders o\nINNER JOIN customers c ON c.id = o.customer_id\nINNER JOIN warehouses w ON w.id = o.warehouse_id\nLEFT JOIN shipments sh ON sh.order_id = o.id\nWHERE o.status IN ('released', 'allocated', 'picking')\n AND sh.id IS NULL\nORDER BY o.required_ship_date, o.priority_code DESC, o.order_number;" + } +] diff --git a/samples/fulfillment-hub/schema.sql b/samples/fulfillment-hub/schema.sql new file mode 100644 index 00000000..db946be8 --- /dev/null +++ b/samples/fulfillment-hub/schema.sql @@ -0,0 +1,452 @@ +-- Fulfillment Hub +-- Fixed snapshot dated 2026-04-24. +-- Highlights: tables, views, indexes, triggers, procedures, saved queries, +-- forms, reports, pipelines, collections, and full-text search layered in +-- by the runnable sample seeder. + +CREATE TABLE customers ( + id INTEGER PRIMARY KEY, + customer_code TEXT COLLATE NOCASE NOT NULL, + name TEXT COLLATE NOCASE NOT NULL, + tier TEXT NOT NULL, + region TEXT NOT NULL, + email TEXT COLLATE NOCASE NOT NULL, + phone TEXT, + is_priority INTEGER NOT NULL +); + +CREATE TABLE suppliers ( + id INTEGER PRIMARY KEY, + supplier_code TEXT COLLATE NOCASE NOT NULL, + name TEXT COLLATE NOCASE NOT NULL, + contact_name TEXT NOT NULL, + email TEXT COLLATE NOCASE NOT NULL, + lead_time_days INTEGER NOT NULL, + status TEXT NOT NULL +); + +CREATE TABLE warehouses ( + id INTEGER PRIMARY KEY, + warehouse_code TEXT COLLATE NOCASE NOT NULL, + name TEXT NOT NULL, + city TEXT NOT NULL, + region TEXT NOT NULL, + is_default INTEGER NOT NULL +); + +CREATE TABLE carriers ( + id INTEGER PRIMARY KEY, + carrier_code TEXT COLLATE NOCASE NOT NULL, + name TEXT NOT NULL, + service_level TEXT NOT NULL +); + +CREATE TABLE products ( + id INTEGER PRIMARY KEY, + sku TEXT COLLATE NOCASE NOT NULL, + name TEXT COLLATE NOCASE NOT NULL, + category TEXT NOT NULL, + description TEXT NOT NULL, + preferred_supplier_id INTEGER NOT NULL REFERENCES suppliers(id), + reorder_point INTEGER NOT NULL, + standard_cost REAL NOT NULL, + sale_price REAL NOT NULL, + is_active INTEGER NOT NULL +); + +CREATE TABLE inventory_positions ( + id INTEGER PRIMARY KEY, + warehouse_id INTEGER NOT NULL REFERENCES warehouses(id), + product_id INTEGER NOT NULL REFERENCES products(id), + on_hand_qty INTEGER NOT NULL, + allocated_qty INTEGER NOT NULL, + inbound_qty INTEGER NOT NULL, + cycle_count_date TEXT NOT NULL +); + +CREATE TABLE purchase_orders ( + id INTEGER PRIMARY KEY, + po_number TEXT COLLATE NOCASE NOT NULL, + supplier_id INTEGER NOT NULL REFERENCES suppliers(id), + warehouse_id INTEGER NOT NULL REFERENCES warehouses(id), + ordered_date TEXT NOT NULL, + expected_date TEXT NOT NULL, + status TEXT NOT NULL, + buyer_name TEXT NOT NULL, + priority_receiving INTEGER NOT NULL, + notes TEXT +); + +CREATE TABLE purchase_order_lines ( + id INTEGER PRIMARY KEY IDENTITY, + purchase_order_id INTEGER NOT NULL REFERENCES purchase_orders(id) ON DELETE CASCADE, + product_id INTEGER NOT NULL REFERENCES products(id), + ordered_qty INTEGER NOT NULL, + received_qty INTEGER NOT NULL, + unit_cost REAL NOT NULL +); + +CREATE TABLE orders ( + id INTEGER PRIMARY KEY, + order_number TEXT COLLATE NOCASE NOT NULL, + customer_id INTEGER NOT NULL REFERENCES customers(id), + warehouse_id INTEGER NOT NULL REFERENCES warehouses(id), + order_date TEXT NOT NULL, + required_ship_date TEXT NOT NULL, + status TEXT NOT NULL, + channel TEXT NOT NULL, + priority_code TEXT NOT NULL, + is_expedited INTEGER NOT NULL, + total_amount REAL NOT NULL, + notes TEXT +); + +CREATE TABLE order_lines ( + id INTEGER PRIMARY KEY IDENTITY, + order_id INTEGER NOT NULL REFERENCES orders(id) ON DELETE CASCADE, + line_number INTEGER NOT NULL, + product_id INTEGER NOT NULL REFERENCES products(id), + ordered_qty INTEGER NOT NULL, + allocated_qty INTEGER NOT NULL, + shipped_qty INTEGER NOT NULL, + unit_price REAL NOT NULL, + line_total REAL NOT NULL +); + +CREATE TABLE shipments ( + id INTEGER PRIMARY KEY, + shipment_number TEXT COLLATE NOCASE NOT NULL, + order_id INTEGER NOT NULL REFERENCES orders(id), + carrier_id INTEGER NOT NULL REFERENCES carriers(id), + warehouse_id INTEGER NOT NULL REFERENCES warehouses(id), + shipped_date TEXT NOT NULL, + status TEXT NOT NULL, + tracking_number TEXT, + picked_by TEXT NOT NULL, + packed_by TEXT NOT NULL +); + +CREATE TABLE shipment_lines ( + id INTEGER PRIMARY KEY IDENTITY, + shipment_id INTEGER NOT NULL REFERENCES shipments(id) ON DELETE CASCADE, + order_line_id INTEGER NOT NULL REFERENCES order_lines(id), + product_id INTEGER NOT NULL REFERENCES products(id), + quantity_shipped INTEGER NOT NULL, + line_total REAL NOT NULL +); + +CREATE TABLE returns ( + id INTEGER PRIMARY KEY, + return_number TEXT COLLATE NOCASE NOT NULL, + order_id INTEGER NOT NULL REFERENCES orders(id), + product_id INTEGER NOT NULL REFERENCES products(id), + warehouse_id INTEGER NOT NULL REFERENCES warehouses(id), + requested_date TEXT NOT NULL, + received_date TEXT, + quantity INTEGER NOT NULL, + status TEXT NOT NULL, + reason TEXT NOT NULL, + disposition TEXT NOT NULL, + requires_qc INTEGER NOT NULL, + notes TEXT +); + +CREATE TABLE ops_playbooks ( + id INTEGER PRIMARY KEY, + title TEXT COLLATE NOCASE NOT NULL, + body TEXT NOT NULL, + tags TEXT NOT NULL, + owner_team TEXT NOT NULL, + updated_date TEXT NOT NULL +); + +CREATE TABLE ops_events ( + id INTEGER PRIMARY KEY IDENTITY, + entity_type TEXT NOT NULL, + entity_id INTEGER NOT NULL, + event_type TEXT NOT NULL, + event_date TEXT NOT NULL, + actor_name TEXT NOT NULL, + details TEXT NOT NULL +); + +CREATE TABLE supplier_receipts_stage ( + id INTEGER PRIMARY KEY IDENTITY, + receipt_batch TEXT NOT NULL, + supplier_code TEXT NOT NULL, + warehouse_code TEXT NOT NULL, + sku TEXT NOT NULL, + received_qty INTEGER NOT NULL, + unit_cost REAL NOT NULL, + received_date TEXT NOT NULL, + reference_number TEXT NOT NULL, + ingest_status TEXT NOT NULL +); + +CREATE TABLE marketplace_orders_stage ( + id INTEGER PRIMARY KEY IDENTITY, + external_order_id TEXT NOT NULL, + customer_email TEXT NOT NULL, + customer_name TEXT NOT NULL, + warehouse_code TEXT NOT NULL, + order_date TEXT NOT NULL, + sku TEXT NOT NULL, + quantity INTEGER NOT NULL, + unit_price REAL NOT NULL, + sales_channel TEXT NOT NULL, + import_status TEXT NOT NULL, + channel_snapshot TEXT +); + +CREATE UNIQUE INDEX idx_customers_code_unique ON customers (customer_code); +CREATE UNIQUE INDEX idx_customers_email_unique ON customers (email); +CREATE UNIQUE INDEX idx_suppliers_code_unique ON suppliers (supplier_code); +CREATE UNIQUE INDEX idx_warehouses_code_unique ON warehouses (warehouse_code); +CREATE UNIQUE INDEX idx_carriers_code_unique ON carriers (carrier_code); +CREATE UNIQUE INDEX idx_products_sku_unique ON products (sku); +CREATE UNIQUE INDEX idx_inventory_positions_unique ON inventory_positions (warehouse_id, product_id); +CREATE UNIQUE INDEX idx_purchase_orders_number_unique ON purchase_orders (po_number); +CREATE UNIQUE INDEX idx_orders_number_unique ON orders (order_number); +CREATE UNIQUE INDEX idx_order_lines_order_line_unique ON order_lines (order_id, line_number); +CREATE UNIQUE INDEX idx_shipments_number_unique ON shipments (shipment_number); +CREATE UNIQUE INDEX idx_returns_number_unique ON returns (return_number); + +CREATE INDEX idx_products_supplier_category ON products (preferred_supplier_id, category); +CREATE INDEX idx_inventory_positions_watch ON inventory_positions (warehouse_id, product_id, on_hand_qty, allocated_qty, inbound_qty); +CREATE INDEX idx_purchase_orders_status_expected ON purchase_orders (status, expected_date); +CREATE INDEX idx_purchase_order_lines_purchase_order ON purchase_order_lines (purchase_order_id, product_id); +CREATE INDEX idx_orders_status_required ON orders (status, required_ship_date, priority_code); +CREATE INDEX idx_orders_customer_date ON orders (customer_id, order_date); +CREATE INDEX idx_order_lines_product ON order_lines (product_id, order_id); +CREATE INDEX idx_shipments_order_status ON shipments (order_id, status, shipped_date); +CREATE INDEX idx_returns_status_requested ON returns (status, requested_date); +CREATE INDEX idx_ops_events_entity_date ON ops_events (entity_type, entity_id, event_date); +CREATE INDEX idx_supplier_receipts_stage_lookup ON supplier_receipts_stage (receipt_batch, sku, reference_number); +CREATE INDEX idx_marketplace_orders_stage_lookup ON marketplace_orders_stage (external_order_id, sku); + +CREATE VIEW order_fulfillment_board AS +SELECT + o.id AS order_id, + o.order_number, + c.customer_code, + c.name AS customer_name, + w.warehouse_code, + o.order_date, + o.required_ship_date, + o.status AS order_status, + o.channel, + o.priority_code, + o.is_expedited, + o.total_amount, + o.notes +FROM orders o +INNER JOIN customers c ON c.id = o.customer_id +INNER JOIN warehouses w ON w.id = o.warehouse_id; + +CREATE VIEW low_stock_watch AS +SELECT + ip.id AS inventory_position_id, + w.id AS warehouse_id, + w.warehouse_code, + w.name AS warehouse_name, + p.id AS product_id, + p.sku, + p.name AS product_name, + p.category, + s.name AS supplier_name, + ip.on_hand_qty, + ip.allocated_qty, + ip.inbound_qty, + ip.on_hand_qty - ip.allocated_qty AS available_qty, + p.reorder_point, + p.reorder_point - ip.on_hand_qty + ip.allocated_qty - ip.inbound_qty AS shortage_qty, + ip.cycle_count_date +FROM inventory_positions ip +INNER JOIN warehouses w ON w.id = ip.warehouse_id +INNER JOIN products p ON p.id = ip.product_id +INNER JOIN suppliers s ON s.id = p.preferred_supplier_id; + +CREATE VIEW purchase_order_receiving_board AS +SELECT + po.id AS purchase_order_id, + po.po_number, + s.name AS supplier_name, + w.warehouse_code, + po.ordered_date, + po.expected_date, + po.status AS po_status, + po.priority_receiving, + p.sku, + p.name AS product_name, + pol.ordered_qty, + pol.received_qty, + pol.ordered_qty - pol.received_qty AS outstanding_qty, + pol.unit_cost +FROM purchase_orders po +INNER JOIN suppliers s ON s.id = po.supplier_id +INNER JOIN warehouses w ON w.id = po.warehouse_id +INNER JOIN purchase_order_lines pol ON pol.purchase_order_id = po.id +INNER JOIN products p ON p.id = pol.product_id; + +CREATE VIEW shipment_manifest_report_source AS +SELECT + sh.id AS shipment_id, + sh.shipment_number, + sh.status AS shipment_status, + sh.shipped_date, + sh.tracking_number, + c2.name AS carrier_name, + o.order_number, + cu.name AS customer_name, + w.warehouse_code, + p.sku, + p.name AS product_name, + sl.quantity_shipped, + sl.line_total +FROM shipments sh +INNER JOIN carriers c2 ON c2.id = sh.carrier_id +INNER JOIN orders o ON o.id = sh.order_id +INNER JOIN customers cu ON cu.id = o.customer_id +INNER JOIN warehouses w ON w.id = sh.warehouse_id +INNER JOIN shipment_lines sl ON sl.shipment_id = sh.id +INNER JOIN products p ON p.id = sl.product_id; + +CREATE VIEW return_queue AS +SELECT + r.id AS return_id, + r.return_number, + r.status AS return_status, + r.requested_date, + r.received_date, + r.reason, + r.disposition, + r.requires_qc, + o.order_number, + cu.name AS customer_name, + p.sku, + p.name AS product_name, + w.warehouse_code, + r.quantity, + r.notes +FROM returns r +INNER JOIN orders o ON o.id = r.order_id +INNER JOIN customers cu ON cu.id = o.customer_id +INNER JOIN products p ON p.id = r.product_id +INNER JOIN warehouses w ON w.id = r.warehouse_id; + +CREATE TRIGGER trg_orders_insert_event AFTER INSERT ON orders +BEGIN + INSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details) + VALUES ('order', NEW.id, 'created', NEW.order_date, 'order-router', 'Sales order created.'); +END; + +CREATE TRIGGER trg_purchase_orders_status_event AFTER UPDATE ON purchase_orders +BEGIN + INSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details) + VALUES ('purchase_order', NEW.id, 'status_changed', NEW.expected_date, 'receiving-console', 'Purchase order status updated.'); +END; + +CREATE TRIGGER trg_shipments_insert_event AFTER INSERT ON shipments +BEGIN + INSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details) + VALUES ('shipment', NEW.id, 'created', NEW.shipped_date, NEW.picked_by, 'Shipment record created.'); +END; + +CREATE TRIGGER trg_returns_insert_event AFTER INSERT ON returns +BEGIN + INSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details) + VALUES ('return', NEW.id, 'requested', NEW.requested_date, 'returns-portal', 'Return request recorded.'); +END; + +INSERT INTO customers VALUES (1001, 'ALP-001', 'Alpine Outdoor', 'Enterprise', 'West', 'ops@alpineoutdoor.example', '206-555-0100', 1); +INSERT INTO customers VALUES (1002, 'BEA-014', 'Beacon Health', 'Strategic', 'Mountain', 'supply@beaconhealth.example', '303-555-0140', 1); +INSERT INTO customers VALUES (1003, 'CED-222', 'Cedar Robotics', 'Growth', 'Central', 'warehouse@cedarrobotics.example', '312-555-0122', 0); +INSERT INTO customers VALUES (1004, 'DRI-090', 'Drift Coffee', 'Growth', 'West', 'buyers@driftcoffee.example', '415-555-0190', 0); +INSERT INTO customers VALUES (1005, 'ELM-119', 'Elm Learning', 'Midmarket', 'Southeast', 'campus@elmlearning.example', '404-555-0119', 0); + +INSERT INTO suppliers VALUES (201, 'NOR-01', 'North Ridge Supply', 'Tara Knox', 'tara@northridge.example', 5, 'active'); +INSERT INTO suppliers VALUES (202, 'PAC-07', 'Pacific Assembly', 'Jon Reeve', 'jon@pacificassembly.example', 7, 'active'); +INSERT INTO suppliers VALUES (203, 'GLA-10', 'Glacier Print Systems', 'Monica Yu', 'monica@glacierprint.example', 6, 'active'); +INSERT INTO suppliers VALUES (204, 'DEL-11', 'Delta Safety Goods', 'Luis Vega', 'luis@deltasafety.example', 4, 'active'); + +INSERT INTO warehouses VALUES (1, 'SEA-FC', 'Seattle Fulfillment Center', 'Seattle', 'West', 1); +INSERT INTO warehouses VALUES (2, 'DEN-DC', 'Denver Distribution Center', 'Denver', 'Mountain', 0); +INSERT INTO warehouses VALUES (3, 'ATL-RTN', 'Atlanta Returns Hub', 'Atlanta', 'Southeast', 0); + +INSERT INTO carriers VALUES (1, 'UPS', 'United Parcel Service', 'Ground'); +INSERT INTO carriers VALUES (2, 'FDX', 'FedEx', '2Day'); +INSERT INTO carriers VALUES (3, 'RLF', 'Regional Line Freight', 'LTL'); + +INSERT INTO products VALUES (501, 'TRJ-100', 'Trail Jacket', 'Apparel', 'Weatherproof trail jacket with packable hood.', 201, 20, 48.50, 89.00, 1); +INSERT INTO products VALUES (502, 'BOT-220', 'Insulated Bottle', 'Gear', 'Double-wall insulated bottle for route drivers.', 201, 25, 11.75, 24.00, 1); +INSERT INTO products VALUES (503, 'RFS-310', 'RFID Scanner', 'Hardware', 'Handheld RFID scanner for pick-pack workflows.', 202, 10, 189.00, 270.00, 1); +INSERT INTO products VALUES (504, 'TAP-410', 'Packing Tape', 'Consumables', 'Heavy-duty tape roll used in every outbound pack station.', 201, 40, 2.30, 6.50, 1); +INSERT INTO products VALUES (505, 'MPR-510', 'Mobile Printer', 'Hardware', 'Portable printer for mobile receiving and cycle counts.', 203, 8, 229.00, 255.00, 1); +INSERT INTO products VALUES (506, 'HZL-610', 'Hazmat Label Kit', 'Safety', 'Hazmat label starter kit for regulated outbound shipments.', 204, 6, 6.25, 10.50, 1); +INSERT INTO products VALUES (507, 'BIN-710', 'Return Bin Large', 'Returns', 'Large bin used in return inspection and refurbishment lanes.', 204, 10, 12.00, 18.80, 1); +INSERT INTO products VALUES (508, 'CCS-810', 'Cold Chain Sensor', 'IoT', 'Temperature sensor used in cold chain shipments and alerts.', 202, 5, 38.50, 48.30, 1); + +INSERT INTO inventory_positions VALUES (1, 1, 501, 18, 6, 18, '2026-04-18'); +INSERT INTO inventory_positions VALUES (2, 1, 502, 64, 8, 0, '2026-04-17'); +INSERT INTO inventory_positions VALUES (3, 1, 503, 9, 4, 10, '2026-04-19'); +INSERT INTO inventory_positions VALUES (4, 1, 504, 140, 10, 50, '2026-04-16'); +INSERT INTO inventory_positions VALUES (5, 2, 501, 7, 2, 10, '2026-04-15'); +INSERT INTO inventory_positions VALUES (6, 2, 505, 4, 1, 8, '2026-04-18'); +INSERT INTO inventory_positions VALUES (7, 2, 506, 2, 0, 20, '2026-04-14'); +INSERT INTO inventory_positions VALUES (8, 2, 507, 21, 3, 0, '2026-04-18'); +INSERT INTO inventory_positions VALUES (9, 3, 503, 5, 0, 3, '2026-04-20'); +INSERT INTO inventory_positions VALUES (10, 3, 507, 44, 2, 25, '2026-04-19'); +INSERT INTO inventory_positions VALUES (11, 3, 508, 3, 1, 6, '2026-04-21'); + +INSERT INTO purchase_orders VALUES (9001, 'PO-9001', 201, 1, '2026-04-18', '2026-04-25', 'open', 'Nina Shah', 1, 'Expedite trail jacket replenishment before weekend promotion.'); +INSERT INTO purchase_orders VALUES (9002, 'PO-9002', 203, 2, '2026-04-16', '2026-04-23', 'open', 'Marco Bell', 0, 'Printer allocation for Denver receiving team.'); +INSERT INTO purchase_orders VALUES (9003, 'PO-9003', 204, 3, '2026-04-20', '2026-04-28', 'open', 'Inez Park', 1, 'Returns lane safety restock.'); + +INSERT INTO purchase_order_lines VALUES (1, 9001, 501, 18, 0, 48.50); +INSERT INTO purchase_order_lines VALUES (2, 9001, 503, 10, 0, 189.00); +INSERT INTO purchase_order_lines VALUES (3, 9002, 505, 8, 3, 229.00); +INSERT INTO purchase_order_lines VALUES (4, 9002, 508, 6, 2, 38.50); +INSERT INTO purchase_order_lines VALUES (5, 9003, 506, 20, 0, 6.25); +INSERT INTO purchase_order_lines VALUES (6, 9003, 507, 25, 0, 12.00); + +INSERT INTO orders VALUES (7001, 'SO-7001', 1001, 1, '2026-04-21', '2026-04-24', 'allocated', 'web', 'high', 1, 626.00, 'Priority launch order for west coast event inventory.'); +INSERT INTO orders VALUES (7002, 'SO-7002', 1002, 2, '2026-04-22', '2026-04-25', 'released', 'edi', 'medium', 0, 496.50, 'Hospital replenishment order awaiting shipment creation.'); +INSERT INTO orders VALUES (7003, 'SO-7003', 1004, 1, '2026-04-20', '2026-04-22', 'shipped', 'marketplace', 'high', 1, 458.00, 'Marketplace order that already shipped from Seattle.'); +INSERT INTO orders VALUES (7004, 'SO-7004', 1003, 3, '2026-04-23', '2026-04-27', 'picking', 'rest-api', 'medium', 0, 307.60, 'Robotics demo order being picked in Atlanta.'); +INSERT INTO orders VALUES (7005, 'SO-7005', 1005, 1, '2026-04-24', '2026-04-29', 'released', 'grpc', 'low', 0, 457.00, 'Campus store replenishment queued for allocation.'); + +INSERT INTO order_lines VALUES (1, 7001, 1, 501, 4, 4, 0, 89.00, 356.00); +INSERT INTO order_lines VALUES (2, 7001, 2, 503, 1, 1, 0, 270.00, 270.00); +INSERT INTO order_lines VALUES (3, 7002, 1, 505, 1, 1, 0, 255.00, 255.00); +INSERT INTO order_lines VALUES (4, 7002, 2, 508, 5, 0, 0, 48.30, 241.50); +INSERT INTO order_lines VALUES (5, 7003, 1, 502, 8, 8, 8, 24.00, 192.00); +INSERT INTO order_lines VALUES (6, 7003, 2, 504, 12, 12, 12, 6.50, 78.00); +INSERT INTO order_lines VALUES (7, 7003, 3, 507, 10, 10, 10, 18.80, 188.00); +INSERT INTO order_lines VALUES (8, 7004, 1, 503, 1, 1, 0, 270.00, 270.00); +INSERT INTO order_lines VALUES (9, 7004, 2, 507, 2, 2, 0, 18.80, 37.60); +INSERT INTO order_lines VALUES (10, 7005, 1, 502, 10, 0, 0, 24.00, 240.00); +INSERT INTO order_lines VALUES (11, 7005, 2, 504, 6, 0, 0, 6.50, 39.00); +INSERT INTO order_lines VALUES (12, 7005, 3, 501, 2, 0, 0, 89.00, 178.00); + +INSERT INTO shipments VALUES (8001, 'SHP-8001', 7003, 1, 1, '2026-04-22', 'shipped', '1Z999AA10123456784', 'Priya Mendez', 'Jordan Hale'); + +INSERT INTO shipment_lines VALUES (1, 8001, 5, 502, 8, 192.00); +INSERT INTO shipment_lines VALUES (2, 8001, 6, 504, 12, 78.00); +INSERT INTO shipment_lines VALUES (3, 8001, 7, 507, 10, 188.00); + +INSERT INTO returns VALUES (8501, 'RMA-8501', 7003, 502, 3, '2026-04-23', '2026-04-24', 2, 'received', 'Damaged insulation cap', 'inspect', 1, 'Inspect before restock because the cap seal failed in transit.'); +INSERT INTO returns VALUES (8502, 'RMA-8502', 7001, 501, 1, '2026-04-24', NULL, 1, 'requested', 'Size swap', 'exchange', 0, 'Customer requested a different size before shipment cut-off.'); + +INSERT INTO ops_playbooks VALUES (1, 'Hazmat Partial Receipt Runbook', 'When a hazmat label kit receipt lands short, stage the pallet, log the partial receipt, and notify the safety lead before allocation resumes.', 'receiving,hazmat,partial', 'Warehouse Ops', '2026-04-18'); +INSERT INTO ops_playbooks VALUES (2, 'Cold Chain Sensor Escalation', 'If a cold chain sensor arrives below reorder coverage, move the open order queue to manual review and issue a supplier escalation within one hour.', 'cold-chain,inventory,escalation', 'Inventory Control', '2026-04-20'); +INSERT INTO ops_playbooks VALUES (3, 'Returns QC Exception Flow', 'Use the return intake form, assign inspection status, and restock only after QC clears any damaged bottle or electronics return.', 'returns,qc,restock', 'Returns Ops', '2026-04-22'); + +INSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details) +VALUES ('inventory_position', 7, 'cycle_count_alert', '2026-04-24', 'inventory-bot', 'Hazmat labels fell below reorder threshold during morning audit.'); + +INSERT INTO ops_events (entity_type, entity_id, event_type, event_date, actor_name, details) +VALUES ('inventory_position', 11, 'cold_chain_watch', '2026-04-24', 'inventory-bot', 'Cold chain sensors require transfer or rush replenishment.'); + +UPDATE purchase_orders +SET status = 'partial' +WHERE id = 9002; diff --git a/scripts/Publish-CSharpDbDaemonRelease.ps1 b/scripts/Publish-CSharpDbDaemonRelease.ps1 new file mode 100644 index 00000000..d1298840 --- /dev/null +++ b/scripts/Publish-CSharpDbDaemonRelease.ps1 @@ -0,0 +1,200 @@ +<# +.SYNOPSIS +Publishes release archives for CSharpDB.Daemon. + +.DESCRIPTION +Publishes CSharpDB.Daemon as self-contained, single-file, non-trimmed +artifacts for one or more runtime identifiers. Each publish output is staged +with the service installation assets under deploy/daemon and then archived for +GitHub Releases. A SHA256SUMS.txt file is generated for all archives in the +archive output folder. + +.PARAMETER Version +The release version used in archive names. When omitted, the script tries to +derive it from GITHUB_REF_NAME when it is a v-prefixed tag, then falls back to +0.0.0-local. + +.PARAMETER Runtime +Runtime identifiers to publish. Defaults to win-x64, linux-x64, and osx-arm64. + +.PARAMETER Configuration +The build configuration passed to dotnet publish. Defaults to Release. + +.PARAMETER OutputRoot +The root folder where publish, stage, and archive outputs are written. Defaults +to artifacts/daemon-release. + +.PARAMETER NoRestore +Passes --no-restore to dotnet publish. + +.EXAMPLE +.\scripts\Publish-CSharpDbDaemonRelease.ps1 -Version 3.4.0 -Runtime win-x64 + +Publishes and archives the Windows x64 daemon release artifact. +#> +[CmdletBinding(SupportsShouldProcess = $true)] +param( + [string]$Version, + [string[]]$Runtime = @('win-x64', 'linux-x64', 'osx-arm64'), + [string]$Configuration = 'Release', + [string]$OutputRoot, + [switch]$NoRestore +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +function Resolve-Version { + param([string]$RequestedVersion) + + if (-not [string]::IsNullOrWhiteSpace($RequestedVersion)) { + return $RequestedVersion.TrimStart('v') + } + + if (-not [string]::IsNullOrWhiteSpace($env:GITHUB_REF_NAME) -and $env:GITHUB_REF_NAME.StartsWith('v')) { + return $env:GITHUB_REF_NAME.Substring(1) + } + + return '0.0.0-local' +} + +function Get-ArchiveName { + param( + [string]$ReleaseVersion, + [string]$Rid + ) + + $extension = if ($Rid.StartsWith('win-', [StringComparison]::OrdinalIgnoreCase)) { 'zip' } else { 'tar.gz' } + return "csharpdb-daemon-v$ReleaseVersion-$Rid.$extension" +} + +function New-TarGzArchive { + param( + [string]$SourceDirectory, + [string]$DestinationPath + ) + + Push-Location $SourceDirectory + try { + & tar -czf $DestinationPath . + if ($LASTEXITCODE -ne 0) { + throw "tar failed with exit code $LASTEXITCODE." + } + } + finally { + Pop-Location + } +} + +$scriptRoot = Split-Path -Parent $MyInvocation.MyCommand.Path +$repoRoot = (Resolve-Path (Join-Path $scriptRoot '..')).Path +$releaseVersion = Resolve-Version $Version + +if ([string]::IsNullOrWhiteSpace($OutputRoot)) { + $OutputRoot = Join-Path $repoRoot 'artifacts\daemon-release' +} +elseif (-not [System.IO.Path]::IsPathRooted($OutputRoot)) { + $OutputRoot = Join-Path $repoRoot $OutputRoot +} + +$projectPath = Join-Path $repoRoot 'src\CSharpDB.Daemon\CSharpDB.Daemon.csproj' +$serviceAssetsRoot = Join-Path $repoRoot 'deploy\daemon' +$publishRoot = Join-Path $OutputRoot 'publish' +$stageRoot = Join-Path $OutputRoot 'stage' +$archiveRoot = Join-Path $OutputRoot 'archives' + +if (-not (Test-Path -Path $projectPath)) { + throw "Daemon project file was not found: $projectPath" +} + +if (-not (Test-Path -Path $serviceAssetsRoot)) { + throw "Service assets folder was not found: $serviceAssetsRoot" +} + +New-Item -ItemType Directory -Path $publishRoot, $stageRoot, $archiveRoot -Force | Out-Null + +$createdArchives = [System.Collections.Generic.List[string]]::new() + +foreach ($rid in $Runtime) { + if ([string]::IsNullOrWhiteSpace($rid)) { + continue + } + + $rid = $rid.Trim() + $publishDir = Join-Path $publishRoot $rid + $stageDir = Join-Path $stageRoot $rid + $archiveName = Get-ArchiveName -ReleaseVersion $releaseVersion -Rid $rid + $archivePath = Join-Path $archiveRoot $archiveName + + if (Test-Path -Path $publishDir) { + Remove-Item -LiteralPath $publishDir -Recurse -Force + } + + if (Test-Path -Path $stageDir) { + Remove-Item -LiteralPath $stageDir -Recurse -Force + } + + $publishArgs = @( + 'publish', + $projectPath, + '--configuration', $Configuration, + '--runtime', $rid, + '--self-contained', 'true', + '--output', $publishDir, + '-p:PublishSingleFile=true', + '-p:PublishTrimmed=false', + '-p:IncludeNativeLibrariesForSelfExtract=true' + ) + + if ($NoRestore.IsPresent) { + $publishArgs += '--no-restore' + } + + if ($PSCmdlet.ShouldProcess($rid, "dotnet $($publishArgs -join ' ')")) { + Write-Host "Publishing CSharpDB.Daemon for $rid..." + & dotnet @publishArgs + + if ($LASTEXITCODE -ne 0) { + throw "dotnet publish failed for $rid with exit code $LASTEXITCODE." + } + + New-Item -ItemType Directory -Path $stageDir -Force | Out-Null + Copy-Item -Path (Join-Path $publishDir '*') -Destination $stageDir -Recurse -Force + Copy-Item -Path $serviceAssetsRoot -Destination (Join-Path $stageDir 'service') -Recurse -Force + Copy-Item -Path (Join-Path $repoRoot 'src\CSharpDB.Daemon\README.md') -Destination (Join-Path $stageDir 'README.md') -Force + + if (Test-Path -Path $archivePath) { + Remove-Item -LiteralPath $archivePath -Force + } + + Write-Host "Creating $archiveName..." + if ($archiveName.EndsWith('.zip', [StringComparison]::OrdinalIgnoreCase)) { + Compress-Archive -Path (Join-Path $stageDir '*') -DestinationPath $archivePath -Force + } + else { + New-TarGzArchive -SourceDirectory $stageDir -DestinationPath $archivePath + } + + $createdArchives.Add($archivePath) | Out-Null + } +} + +$archives = Get-ChildItem -Path $archiveRoot -File | + Where-Object { $_.Name -like 'csharpdb-daemon-v*.zip' -or $_.Name -like 'csharpdb-daemon-v*.tar.gz' } | + Sort-Object Name + +$checksumsPath = Join-Path $archiveRoot 'SHA256SUMS.txt' +$checksumLines = foreach ($archive in $archives) { + $hash = Get-FileHash -Path $archive.FullName -Algorithm SHA256 + "$($hash.Hash.ToLowerInvariant()) $($archive.Name)" +} + +Set-Content -Path $checksumsPath -Value $checksumLines -Encoding ASCII + +Write-Host "Daemon release archives complete." +Write-Host " Version: $releaseVersion" +Write-Host " Archive root: $archiveRoot" +foreach ($archive in $createdArchives) { + Write-Host " Created: $archive" +} +Write-Host " Checksums: $checksumsPath" diff --git a/scripts/README.md b/scripts/README.md index 9e32449f..25f86829 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,11 +1,104 @@ # Scripts -These scripts are developer and operator helpers for local source runs and -repository maintenance tasks. +These scripts are developer, operator, and release helpers for local source +runs, repository maintenance, and CSharpDB daemon release packaging. -They do not install Windows services, `systemd` units, or scheduled tasks. They -launch `dotnet run` processes from the repo and update the admin app config so -the web UI starts in the expected transport mode. +The local start scripts launch `dotnet run` processes from the repo and update +the admin app config so the web UI starts in the expected transport mode. +Daemon service install scripts live under [`deploy/daemon`](../deploy/daemon) +and are included in daemon release archives. + +## How The Scripts Fit The Release Cycle + +There are three different script groups: + +- Release maintainers use `scripts/Publish-CSharpDbDaemonRelease.ps1` directly + for local packaging checks. The GitHub Release workflow also uses this script + when a `v*` tag is pushed. +- Operators use the service scripts after a release is published. These scripts + are included inside each daemon archive under `service/`. +- Developers use `Start-CSharpDbAdminAndDaemon.ps1` and + `Start-CSharpDbAdminDirect.ps1` for local source runs. Developers can also + use `Start-CSharpDbAdminFormsWeb.ps1` when they only need the runtime form + host. These are not release packaging scripts. + +## Release Maintainer Walkthrough + +Use this path before tagging or when validating release packaging locally. + +1. Build and test the repo as usual. + +```powershell +dotnet build CSharpDB.slnx -c Release +dotnet test tests\CSharpDB.Daemon.Tests\CSharpDB.Daemon.Tests.csproj -c Release +``` + +2. Publish one local archive for a fast packaging check. + +```powershell +.\scripts\Publish-CSharpDbDaemonRelease.ps1 ` + -Version 3.4.0 ` + -Runtime win-x64 ` + -OutputRoot artifacts\daemon-release-local +``` + +3. Confirm the archive and checksum exist. + +```powershell +Get-ChildItem artifacts\daemon-release-local\archives +Get-Content artifacts\daemon-release-local\archives\SHA256SUMS.txt +``` + +4. For the real release, push a `v*` tag. The GitHub Release workflow publishes + the daemon archives for `win-x64`, `linux-x64`, and `osx-arm64`, smoke-starts + each extracted binary, calls the daemon REST `/api/info` endpoint, verifies a + gRPC `GetInfoAsync` client call, combines checksums, and attaches everything + to the GitHub Release. + +```powershell +git tag v3.4.0 +git push origin v3.4.0 +``` + +## Operator Walkthrough + +Use this path after a GitHub Release is published. + +1. Download the daemon archive for the target OS: + `csharpdb-daemon-v{version}-win-x64.zip`, + `csharpdb-daemon-v{version}-linux-x64.tar.gz`, or + `csharpdb-daemon-v{version}-osx-arm64.tar.gz`. +2. Verify the archive with the published `SHA256SUMS.txt`. +3. Extract the archive on the target machine. +4. Run the matching service installer from inside the extracted archive. +5. Connect REST clients to `http://127.0.0.1:5820/api/...` or gRPC clients to + the same base URL. The installed daemon exposes both transports by default. + +Windows, from an elevated PowerShell session: + +```powershell +.\service\windows\install-csharpdb-daemon.ps1 -Start +``` + +Linux: + +```bash +sudo ./service/linux/install-csharpdb-daemon.sh --start +``` + +macOS: + +```bash +sudo ./service/macos/install-csharpdb-daemon.sh --start +``` + +To upgrade, extract the newer archive and rerun the same installer with the +same data directory plus `-Force` on Windows or `--force` on Linux/macOS. The +installers replace daemon files but do not delete the database directory. + +To disable the daemon REST surface while keeping gRPC enabled, set +`CSharpDB__Daemon__EnableRestApi=false` in the service environment or generated +`appsettings.Production.json`, then restart the service. ## Scripts @@ -46,9 +139,12 @@ What it does: - waits for the daemon port to come up - starts `CSharpDB.Admin` +The daemon started by this script also serves the REST `/api` surface on the +same base URL unless `CSharpDB:Daemon:EnableRestApi` is disabled. + ### `Start-CSharpDbAdminDirect.ps1` -Use this when the admin site should open the database file directly without the +Use this when the admin site should open a local database directly without the daemon. What it does: @@ -59,6 +155,92 @@ What it does: - keep or set `ConnectionStrings:CSharpDB` - starts only `CSharpDB.Admin` +The default Admin direct configuration uses +`CSharpDB:HostDatabase:OpenMode = "HybridIncrementalDurable"`, so direct mode +opens through the hybrid incremental-durable local host path. The Admin host +warms one in-process database instance at startup and keeps it alive until the +Admin app shuts down or the user switches databases. Set +`CSharpDB:HostDatabase:OpenMode = "Direct"` if you need the older plain direct +open path for a local run. + +### `Start-CSharpDbAdminFormsWeb.ps1` + +Use this when you want to run stored forms through the forms-only runtime host +without opening the full Admin studio. + +What it does: + +- reads the default target database path from + [`src/CSharpDB.Admin.Forms.Web/appsettings.json`](../src/CSharpDB.Admin.Forms.Web/appsettings.json) + unless you pass `-DataSource` +- starts `src/CSharpDB.Admin.Forms.Web` +- passes the resolved `CSharpDB:DataSource` and `--urls` values through + command-line configuration overrides +- waits for the forms host port to accept TCP connections +- optionally opens the root runtime page in the default browser + +Use a sample database that already contains seeded forms, such as the +Fulfillment Hub sample database, when you want the runtime root page to list +real forms immediately. + +### `Publish-CSharpDbDaemonRelease.ps1` + +Use this when preparing self-contained daemon release archives. + +What it does: + +- publishes `src/CSharpDB.Daemon` for one or more runtime identifiers +- uses Release, self-contained, single-file, non-trimmed publish settings +- stages the daemon with service assets from `deploy/daemon` +- creates `csharpdb-daemon-v{version}-{rid}.zip` for Windows +- creates `csharpdb-daemon-v{version}-{rid}.tar.gz` for Linux/macOS +- writes `SHA256SUMS.txt` + +Examples: + +```powershell +.\scripts\Publish-CSharpDbDaemonRelease.ps1 -Version 3.4.0 -Runtime win-x64 +.\scripts\Publish-CSharpDbDaemonRelease.ps1 -Version 3.4.0 -Runtime linux-x64,osx-arm64 +``` + +Default runtimes: + +- `win-x64` +- `linux-x64` +- `osx-arm64` + +Outputs are written under `artifacts\daemon-release` unless `-OutputRoot` is +provided. + +## Daemon Service Installers + +Release archives include OS service assets under `service/`: + +- Windows: `service/windows/install-csharpdb-daemon.ps1` +- Windows: `service/windows/uninstall-csharpdb-daemon.ps1` +- Linux: `service/linux/csharpdb-daemon.service` +- Linux: `service/linux/install-csharpdb-daemon.sh` +- Linux: `service/linux/uninstall-csharpdb-daemon.sh` +- macOS: `service/macos/com.csharpdb.daemon.plist` +- macOS: `service/macos/install-csharpdb-daemon.sh` +- macOS: `service/macos/uninstall-csharpdb-daemon.sh` + +Default service settings: + +| Platform | Service | Install directory | Data directory | URL | +|----------|---------|-------------------|----------------|-----| +| Windows | `CSharpDBDaemon` | `C:\Program Files\CSharpDB\Daemon` | `C:\ProgramData\CSharpDB` | `http://127.0.0.1:5820` | +| Linux | `csharpdb-daemon` | `/opt/csharpdb-daemon` | `/var/lib/csharpdb` | `http://127.0.0.1:5820` | +| macOS | `com.csharpdb.daemon` | `/usr/local/lib/csharpdb-daemon` | `/usr/local/var/csharpdb` | `http://127.0.0.1:5820` | + +Installers accept service name, install directory, data directory, bind URL, +and force/overwrite options. Windows scripts support `-WhatIf`; Linux and macOS +scripts require `sudo` and fail early when not run as root. + +The generated production config enables REST by default through +`CSharpDB:Daemon:EnableRestApi=true`. Service-level environment variables can +override this with `CSharpDB__Daemon__EnableRestApi=false`. + ## Quick Start From the repo root in PowerShell: @@ -73,6 +255,14 @@ Start the admin in direct mode: & .\scripts\Start-CSharpDbAdminDirect.ps1 ``` +Start the forms-only runtime host against the Fulfillment Hub sample database: + +```powershell +& .\scripts\Start-CSharpDbAdminFormsWeb.ps1 ` + -DataSource samples\fulfillment-hub\bin\Debug\net10.0\fulfillment-hub-demo.db ` + -OpenBrowser +``` + Open the admin site automatically after startup: ```powershell @@ -200,7 +390,7 @@ If your machine is slow or the first build is cold, increase the wait time: ## Config Sources -The gRPC startup script resolves values from the repo files below: +The admin-and-daemon startup script resolves values from the repo files below: - daemon URL: [`src/CSharpDB.Daemon/Properties/launchSettings.json`](../src/CSharpDB.Daemon/Properties/launchSettings.json) @@ -216,6 +406,13 @@ The direct-mode script resolves values from: - admin database connection string: [`src/CSharpDB.Admin/appsettings.json`](../src/CSharpDB.Admin/appsettings.json), unless you pass `-ConnectionString` +The forms-runtime script resolves values from: + +- forms host database path: + [`src/CSharpDB.Admin.Forms.Web/appsettings.json`](../src/CSharpDB.Admin.Forms.Web/appsettings.json), unless you pass `-DataSource` +- forms host URL: + the script `-Url` parameter, defaulting to `http://127.0.0.1:5095` + ## Use `Get-Help` Both scripts now include comment-based help: @@ -223,6 +420,7 @@ Both scripts now include comment-based help: ```powershell Get-Help .\scripts\Start-CSharpDbAdminAndDaemon.ps1 -Detailed Get-Help .\scripts\Start-CSharpDbAdminDirect.ps1 -Detailed +Get-Help .\scripts\Start-CSharpDbAdminFormsWeb.ps1 -Detailed ``` ## Troubleshooting diff --git a/scripts/Start-CSharpDbAdminFormsWeb.ps1 b/scripts/Start-CSharpDbAdminFormsWeb.ps1 new file mode 100644 index 00000000..d8a92cc2 --- /dev/null +++ b/scripts/Start-CSharpDbAdminFormsWeb.ps1 @@ -0,0 +1,292 @@ +<# +.SYNOPSIS +Starts the forms-only web host against a target CSharpDB database. + +.DESCRIPTION +This script is intended for local development and operator-style runtime +testing. It starts `src/CSharpDB.Admin.Forms.Web`, points it at a direct local +database path, waits for the web host to accept TCP connections, and can open +the runtime root page in the default browser. + +The script does not rewrite project configuration files. It passes the target +database path and bind URL as command-line configuration overrides to +`dotnet run`. + +.PARAMETER DataSource +Overrides the target database path. When omitted, the script uses +`CSharpDB:DataSource` from `src/CSharpDB.Admin.Forms.Web/appsettings.json`, +falling back to `csharpdb.db`. + +.PARAMETER Url +The URL passed to `--urls` for the forms host. Defaults to +`http://127.0.0.1:5095`. + +.PARAMETER NoLaunch +Resolves values but does not start the forms host. + +.PARAMETER OpenBrowser +Opens the forms runtime root page in the default browser after startup +succeeds. + +.PARAMETER PassThru +Returns the resolved URL, database path, and process ID so the caller can stop +the process later with `Stop-Process`. + +.PARAMETER StartupTimeoutSeconds +How long to wait for the forms host endpoint to start accepting TCP +connections. + +.EXAMPLE +powershell -ExecutionPolicy Bypass -File .\scripts\Start-CSharpDbAdminFormsWeb.ps1 + +Starts the forms runtime host using the default data source from +`src/CSharpDB.Admin.Forms.Web/appsettings.json`. + +.EXAMPLE +.\scripts\Start-CSharpDbAdminFormsWeb.ps1 ` + -DataSource C:\Users\maxim\source\Code\CSharpDB\samples\fulfillment-hub\bin\Debug\net10.0\fulfillment-hub-demo.db ` + -OpenBrowser + +Starts the forms runtime host against the Fulfillment Hub sample database and +opens the root page in the default browser. + +.EXAMPLE +$session = & .\scripts\Start-CSharpDbAdminFormsWeb.ps1 -PassThru +Stop-Process -Id $session.FormsPid + +Starts the forms runtime host and captures the process ID so it can be stopped +later. +#> +[CmdletBinding()] +param( + [string]$DataSource, + [string]$Url = 'http://127.0.0.1:5095', + [switch]$NoLaunch, + [switch]$OpenBrowser, + [switch]$PassThru, + [int]$StartupTimeoutSeconds = 30 +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +function Read-JsonFile { + param( + [Parameter(Mandatory = $true)] + [string]$Path + ) + + if (-not (Test-Path -Path $Path)) { + throw "Required file was not found: $Path" + } + + return Get-Content -Path $Path -Raw | ConvertFrom-Json +} + +function Wait-ForTcpEndpoint { + param( + [Parameter(Mandatory = $true)] + [Uri]$Uri, + [Parameter(Mandatory = $true)] + [int]$TimeoutSeconds, + [System.Diagnostics.Process]$Process + ) + + $deadline = (Get-Date).AddSeconds($TimeoutSeconds) + + while ((Get-Date) -lt $deadline) { + if ($null -ne $Process) { + $Process.Refresh() + if ($Process.HasExited) { + return $false + } + } + + $client = $null + + try { + $client = [System.Net.Sockets.TcpClient]::new() + $connectTask = $client.ConnectAsync($Uri.Host, $Uri.Port) + + if ($connectTask.Wait([TimeSpan]::FromSeconds(1)) -and $client.Connected) { + return $true + } + } + catch { + } + finally { + if ($null -ne $client) { + $client.Dispose() + } + } + + Start-Sleep -Milliseconds 500 + } + + return $false +} + +function Test-TcpEndpointAvailable { + param( + [Parameter(Mandatory = $true)] + [Uri]$Uri + ) + + $client = $null + + try { + $client = [System.Net.Sockets.TcpClient]::new() + $connectTask = $client.ConnectAsync($Uri.Host, $Uri.Port) + return -not ($connectTask.Wait([TimeSpan]::FromMilliseconds(400)) -and $client.Connected) + } + catch { + return $true + } + finally { + if ($null -ne $client) { + $client.Dispose() + } + } +} + +function Stop-ProcessIfRunning { + param( + [Parameter(Mandatory = $true)] + [System.Diagnostics.Process]$Process + ) + + $Process.Refresh() + if (-not $Process.HasExited) { + Stop-Process -Id $Process.Id -Force + } +} + +function Get-ConfiguredDataSource { + param( + [Parameter(Mandatory = $true)] + [object]$Config + ) + + if ($null -ne $Config.PSObject.Properties['CSharpDB'] -and + $null -ne $Config.CSharpDB -and + $null -ne $Config.CSharpDB.PSObject.Properties['DataSource'] -and + -not [string]::IsNullOrWhiteSpace($Config.CSharpDB.DataSource)) { + return $Config.CSharpDB.DataSource + } + + if ($null -ne $Config.PSObject.Properties['ConnectionStrings'] -and + $null -ne $Config.ConnectionStrings -and + $null -ne $Config.ConnectionStrings.PSObject.Properties['CSharpDB'] -and + -not [string]::IsNullOrWhiteSpace($Config.ConnectionStrings.CSharpDB)) { + $match = [regex]::Match($Config.ConnectionStrings.CSharpDB, '(?i)\bData Source\s*=\s*(?[^;]+)') + if ($match.Success) { + return $match.Groups['value'].Value.Trim() + } + } + + return 'csharpdb.db' +} + +function Resolve-RepoRelativePath { + param( + [Parameter(Mandatory = $true)] + [string]$PathValue, + [Parameter(Mandatory = $true)] + [string]$RepoRoot + ) + + if ([string]::IsNullOrWhiteSpace($PathValue)) { + return $PathValue + } + + if ([System.IO.Path]::IsPathRooted($PathValue)) { + return $PathValue + } + + return (Join-Path $RepoRoot $PathValue) +} + +$scriptRoot = Split-Path -Parent $MyInvocation.MyCommand.Path +$repoRoot = (Resolve-Path (Join-Path $scriptRoot '..')).Path + +$formsWebProjectPath = Join-Path $repoRoot 'src\CSharpDB.Admin.Forms.Web\CSharpDB.Admin.Forms.Web.csproj' +$formsWebAppSettingsPath = Join-Path $repoRoot 'src\CSharpDB.Admin.Forms.Web\appsettings.json' + +if (-not (Test-Path -Path $formsWebProjectPath)) { + throw "Forms web project file was not found: $formsWebProjectPath" +} + +$formsWebConfig = Read-JsonFile -Path $formsWebAppSettingsPath +$effectiveDataSource = if ([string]::IsNullOrWhiteSpace($DataSource)) { + Get-ConfiguredDataSource -Config $formsWebConfig +} +else { + $DataSource +} + +$resolvedDataSource = Resolve-RepoRelativePath -PathValue $effectiveDataSource -RepoRoot $repoRoot +$urlUri = [Uri]$Url + +if (-not (Test-TcpEndpointAvailable -Uri $urlUri)) { + throw "The forms runtime URL is already accepting TCP connections: $Url" +} + +if (-not (Test-Path -Path $resolvedDataSource)) { + Write-Warning "The target database path does not exist yet: $resolvedDataSource" +} + +$result = [pscustomobject]@{ + FormsUrl = $Url + DataSource = $resolvedDataSource + FormsPid = $null +} + +if ($NoLaunch) { + if ($PassThru) { + return $result + } + + Write-Host "Resolved forms host settings:" + Write-Host " Url: $Url" + Write-Host " DataSource: $resolvedDataSource" + return +} + +$dotnetArgs = @( + 'run', + '--project', $formsWebProjectPath, + '--', + '--urls', $Url, + "--CSharpDB:DataSource=$resolvedDataSource" +) + +Write-Host "Starting CSharpDB.Admin.Forms.Web..." +Write-Host " Url: $Url" +Write-Host " DataSource: $resolvedDataSource" + +$formsProcess = Start-Process -FilePath 'dotnet' -ArgumentList $dotnetArgs -WorkingDirectory $repoRoot -PassThru + +try { + if (-not (Wait-ForTcpEndpoint -Uri $urlUri -TimeoutSeconds $StartupTimeoutSeconds -Process $formsProcess)) { + Stop-ProcessIfRunning -Process $formsProcess + throw "The forms runtime did not start accepting TCP connections within $StartupTimeoutSeconds second(s): $Url" + } +} +catch { + Stop-ProcessIfRunning -Process $formsProcess + throw +} + +Write-Host "CSharpDB.Admin.Forms.Web is running." +Write-Host " PID: $($formsProcess.Id)" +Write-Host " Url: $Url" + +if ($OpenBrowser) { + Start-Process $Url | Out-Null +} + +$result.FormsPid = $formsProcess.Id + +if ($PassThru) { + return $result +} diff --git a/src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj b/src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj new file mode 100644 index 00000000..43dc2a3f --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj @@ -0,0 +1,16 @@ + + + + net10.0 + enable + enable + CSharpDB.Admin.Forms.Web + false + + + + + + + + diff --git a/src/CSharpDB.Admin.Forms.Web/Components/App.razor b/src/CSharpDB.Admin.Forms.Web/Components/App.razor new file mode 100644 index 00000000..e220983a --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/Components/App.razor @@ -0,0 +1,17 @@ + + + + + + + CSharpDB Forms + + + + + + + + + + diff --git a/src/CSharpDB.Admin.Forms.Web/Components/Layout/MainLayout.razor b/src/CSharpDB.Admin.Forms.Web/Components/Layout/MainLayout.razor new file mode 100644 index 00000000..4f3f76da --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/Components/Layout/MainLayout.razor @@ -0,0 +1,3 @@ +@inherits LayoutComponentBase + +@Body diff --git a/src/CSharpDB.Admin.Forms.Web/Components/Pages/FormRuntime.razor b/src/CSharpDB.Admin.Forms.Web/Components/Pages/FormRuntime.razor new file mode 100644 index 00000000..803fd8d2 --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/Components/Pages/FormRuntime.razor @@ -0,0 +1,9 @@ +@page "/forms/{FormId}" + +Form Runtime + + + +@code { + [Parameter] public string FormId { get; set; } = string.Empty; +} diff --git a/src/CSharpDB.Admin.Forms.Web/Components/Pages/Home.razor b/src/CSharpDB.Admin.Forms.Web/Components/Pages/Home.razor new file mode 100644 index 00000000..29b0521b --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/Components/Pages/Home.razor @@ -0,0 +1,86 @@ +@page "/" +@inject IFormRepository FormRepository +@inject ICSharpDbClient DbClient + +Forms Runtime + +
+
+
+

Forms Runtime

+

Runnable form host for @DbClient.DataSource

+
+
+ +
+ This host runs stored form definitions without exposing the designer. Seed forms into the target database, then open them from this list. +
+ + @if (_loading) + { +
Loading forms...
+ } + else if (_error is not null) + { +
@_error
+ } + else if (_forms.Count == 0) + { +
+ No forms were found in this database. Run a sample or seed form metadata into the current database first. +
+ } + else + { +
@_forms.Count form(s) available
+ + + + + + + + + + + @foreach (FormDefinition form in _forms) + { + + + + + + + } + +
NameSourceForm Id
@form.Name@form.TableName@form.FormId + Open +
+ } +
+ +@code { + private IReadOnlyList _forms = []; + private bool _loading = true; + private string? _error; + + protected override async Task OnInitializedAsync() + { + try + { + _forms = (await FormRepository.ListAsync()) + .OrderBy(form => form.Name, StringComparer.OrdinalIgnoreCase) + .ToArray(); + } + catch (Exception ex) + { + _error = $"Failed to load forms: {ex.Message}"; + } + finally + { + _loading = false; + } + } + + private static string GetFormHref(string formId) => $"/forms/{Uri.EscapeDataString(formId)}"; +} diff --git a/src/CSharpDB.Admin.Forms.Web/Components/Routes.razor b/src/CSharpDB.Admin.Forms.Web/Components/Routes.razor new file mode 100644 index 00000000..63e4bcc6 --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/Components/Routes.razor @@ -0,0 +1,18 @@ + + + + + + +
+
+
+

Forms Runtime

+

Route not found.

+
+ All forms +
+
+ + + diff --git a/src/CSharpDB.Admin.Forms.Web/Configuration/FormsHostClientOptionsBuilder.cs b/src/CSharpDB.Admin.Forms.Web/Configuration/FormsHostClientOptionsBuilder.cs new file mode 100644 index 00000000..482bff72 --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/Configuration/FormsHostClientOptionsBuilder.cs @@ -0,0 +1,170 @@ +using CSharpDB.Client; +using CSharpDB.Engine; +using Microsoft.Extensions.Configuration; + +namespace CSharpDB.Admin.Forms.Web.Configuration; + +public enum FormsHostOpenMode +{ + HybridIncrementalDurable = 0, + Direct = 1, +} + +public sealed class FormsHostDatabaseOptions +{ + public FormsHostOpenMode OpenMode { get; init; } = FormsHostOpenMode.HybridIncrementalDurable; + + public ImplicitInsertExecutionMode ImplicitInsertExecutionMode { get; init; } = + ImplicitInsertExecutionMode.ConcurrentWriteTransactions; + + public bool UseWriteOptimizedPreset { get; init; } = true; + + public string[] HotTableNames { get; init; } = []; + + public string[] HotCollectionNames { get; init; } = []; +} + +public static class FormsHostClientOptionsBuilder +{ + private const string FallbackDataSource = "csharpdb.db"; + + public static FormsHostDatabaseOptions BindHostDatabaseOptions(IConfiguration configuration) + { + ArgumentNullException.ThrowIfNull(configuration); + + FormsHostDatabaseOptions bound = configuration.GetSection("CSharpDB:HostDatabase").Get() + ?? new FormsHostDatabaseOptions(); + + return new FormsHostDatabaseOptions + { + OpenMode = bound.OpenMode, + ImplicitInsertExecutionMode = bound.ImplicitInsertExecutionMode, + UseWriteOptimizedPreset = bound.UseWriteOptimizedPreset, + HotTableNames = bound.HotTableNames ?? [], + HotCollectionNames = bound.HotCollectionNames ?? [], + }; + } + + public static CSharpDbClientOptions Build(IConfiguration configuration, FormsHostDatabaseOptions hostDatabaseOptions) + { + ArgumentNullException.ThrowIfNull(configuration); + ArgumentNullException.ThrowIfNull(hostDatabaseOptions); + + string? endpoint = configuration["CSharpDB:Endpoint"]; + string? dataSource = configuration["CSharpDB:DataSource"]; + CSharpDbTransport? transport = ParseTransport(configuration["CSharpDB:Transport"]); + string? connectionString = configuration.GetConnectionString("CSharpDB"); + + if (!string.IsNullOrWhiteSpace(endpoint)) + { + if (transport == CSharpDbTransport.Direct || (transport is null && EndpointLooksLikeDirectPath(endpoint))) + return BuildDirectDataSource(NormalizeDirectPath(endpoint), hostDatabaseOptions); + + return new CSharpDbClientOptions + { + Transport = transport, + Endpoint = endpoint, + }; + } + + if (!string.IsNullOrWhiteSpace(dataSource)) + return BuildDirectDataSource(dataSource, hostDatabaseOptions); + + if (!string.IsNullOrWhiteSpace(connectionString)) + { + return new CSharpDbClientOptions + { + Transport = transport ?? CSharpDbTransport.Direct, + ConnectionString = connectionString, + DirectDatabaseOptions = BuildDirectDatabaseOptions(hostDatabaseOptions), + HybridDatabaseOptions = BuildHybridDatabaseOptionsOrNull(hostDatabaseOptions), + }; + } + + if (transport is not null && transport != CSharpDbTransport.Direct) + return new CSharpDbClientOptions { Transport = transport }; + + return BuildDirectDataSource(FallbackDataSource, hostDatabaseOptions); + } + + public static CSharpDbClientOptions BuildDirectDataSource(string dataSource, FormsHostDatabaseOptions hostDatabaseOptions) + { + ArgumentException.ThrowIfNullOrWhiteSpace(dataSource); + ArgumentNullException.ThrowIfNull(hostDatabaseOptions); + + return new CSharpDbClientOptions + { + Transport = CSharpDbTransport.Direct, + DataSource = dataSource, + DirectDatabaseOptions = BuildDirectDatabaseOptions(hostDatabaseOptions), + HybridDatabaseOptions = BuildHybridDatabaseOptionsOrNull(hostDatabaseOptions), + }; + } + + public static DatabaseOptions BuildDirectDatabaseOptions(FormsHostDatabaseOptions hostDatabaseOptions) + { + ArgumentNullException.ThrowIfNull(hostDatabaseOptions); + + var options = new DatabaseOptions + { + ImplicitInsertExecutionMode = hostDatabaseOptions.ImplicitInsertExecutionMode, + }; + + if (!hostDatabaseOptions.UseWriteOptimizedPreset) + return options; + + return options.ConfigureStorageEngine(builder => builder.UseWriteOptimizedPreset()); + } + + public static HybridDatabaseOptions? BuildHybridDatabaseOptionsOrNull(FormsHostDatabaseOptions hostDatabaseOptions) + { + ArgumentNullException.ThrowIfNull(hostDatabaseOptions); + + if (hostDatabaseOptions.OpenMode != FormsHostOpenMode.HybridIncrementalDurable) + return null; + + return new HybridDatabaseOptions + { + PersistenceMode = HybridPersistenceMode.IncrementalDurable, + HotTableNames = hostDatabaseOptions.HotTableNames, + HotCollectionNames = hostDatabaseOptions.HotCollectionNames, + }; + } + + private static CSharpDbTransport? ParseTransport(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + return null; + + return value.Trim().ToLowerInvariant() switch + { + "direct" => CSharpDbTransport.Direct, + "http" => CSharpDbTransport.Http, + "grpc" => CSharpDbTransport.Grpc, + "namedpipes" => CSharpDbTransport.NamedPipes, + "named-pipes" => CSharpDbTransport.NamedPipes, + "npipe" => CSharpDbTransport.NamedPipes, + "pipe" => CSharpDbTransport.NamedPipes, + _ => throw new InvalidOperationException($"Unsupported transport '{value}'."), + }; + } + + private static bool EndpointLooksLikeDirectPath(string endpoint) + { + if (Uri.TryCreate(endpoint, UriKind.Absolute, out Uri? uri)) + return string.Equals(uri.Scheme, Uri.UriSchemeFile, StringComparison.OrdinalIgnoreCase); + + return !endpoint.Contains("://", StringComparison.Ordinal); + } + + private static string NormalizeDirectPath(string endpoint) + { + if (Uri.TryCreate(endpoint, UriKind.Absolute, out Uri? uri) && + string.Equals(uri.Scheme, Uri.UriSchemeFile, StringComparison.OrdinalIgnoreCase)) + { + return uri.LocalPath; + } + + return endpoint; + } +} diff --git a/src/CSharpDB.Admin.Forms.Web/Program.cs b/src/CSharpDB.Admin.Forms.Web/Program.cs new file mode 100644 index 00000000..29a20ce5 --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/Program.cs @@ -0,0 +1,39 @@ +using CSharpDB.Admin.Forms.Services; +using CSharpDB.Admin.Forms.Web.Components; +using CSharpDB.Admin.Forms.Web.Configuration; +using CSharpDB.Client; + +var builder = WebApplication.CreateBuilder(args); + +builder.Services.AddRazorComponents() + .AddInteractiveServerComponents(); + +builder.Services.AddSingleton(sp => + FormsHostClientOptionsBuilder.BindHostDatabaseOptions(sp.GetRequiredService())); +builder.Services.AddSingleton(sp => +{ + IConfiguration configuration = sp.GetRequiredService(); + FormsHostDatabaseOptions hostDatabaseOptions = sp.GetRequiredService(); + return CSharpDbClient.Create(FormsHostClientOptionsBuilder.Build(configuration, hostDatabaseOptions)); +}); +builder.Services.AddCSharpDbAdminForms(); + +var app = builder.Build(); + +await using (var scope = app.Services.CreateAsyncScope()) +{ + var dbClient = scope.ServiceProvider.GetRequiredService(); + _ = await dbClient.GetInfoAsync(); +} + +if (!app.Environment.IsDevelopment()) + app.UseExceptionHandler("/error"); + +app.MapStaticAssets(); +app.UseStaticFiles(); +app.UseAntiforgery(); + +app.MapRazorComponents() + .AddInteractiveServerRenderMode(); + +app.Run(); diff --git a/src/CSharpDB.Admin.Forms.Web/README.md b/src/CSharpDB.Admin.Forms.Web/README.md new file mode 100644 index 00000000..d39129fd --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/README.md @@ -0,0 +1,133 @@ +# CSharpDB.Admin.Forms.Web + +Blazor Server runtime host for stored CSharpDB forms. + +`CSharpDB.Admin.Forms.Web` is an application host, not a NuGet package. It +loads form definitions from a target CSharpDB database and runs them through +the shared `DataEntry` runtime from `CSharpDB.Admin.Forms` without exposing the +form designer. + +## What This Project Is For + +Use this project when you want to: + +- run stored forms as a focused data-entry web app +- point a simple web host at a database that already contains form metadata +- give operators a runnable form surface without the full Admin studio +- reuse the same stored forms in multiple hosts + +Use `CSharpDB.Admin` when users also need schema browsing, query tabs, +procedures, pipelines, report design, and form design mode. + +## What This Project Provides + +- form list at `/` +- runnable form routes at `/forms/{formId}` +- shared form runtime from `CSharpDB.Admin.Forms` +- record create, update, delete, search, paging, navigation, child grids, and + child tabs +- runtime-only host behavior with no visible `Edit Form` designer action + +## Runtime Model + +The host registers one `ICSharpDbClient` from configuration and then adds the +standard `AddCSharpDbAdminForms()` service set. + +At runtime: + +1. the host opens the configured database through `CSharpDbClient` +2. `IFormRepository` reads stored form definitions from the target database +3. the root page lists available forms +4. `/forms/{formId}` runs the selected form through `Pages/DataEntry.razor` + +The host does not create forms. The target database must already contain them. + +For example, the Fulfillment Hub sample seeds forms into: + +```text +samples/fulfillment-hub/bin/Debug/net10.0/fulfillment-hub-demo.db +``` + +## Configuration + +The host supports the same broad connection shapes as the other CSharpDB hosts: + +- direct local database access through `CSharpDB:DataSource` +- direct access through `ConnectionStrings:CSharpDB` +- remote access through `CSharpDB:Transport` plus `CSharpDB:Endpoint` + +Default `appsettings.json`: + +```json +{ + "CSharpDB": { + "Transport": "Direct", + "DataSource": "csharpdb.db", + "HostDatabase": { + "OpenMode": "HybridIncrementalDurable", + "UseWriteOptimizedPreset": true, + "HotTableNames": [], + "HotCollectionNames": [] + } + } +} +``` + +In direct mode, the host uses the same direct/hybrid database option builder +pattern as Admin, including the write-optimized preset by default. + +## Running Locally + +Default run: + +```powershell +dotnet run --project src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj +``` + +Run against a specific database: + +```powershell +dotnet run --project src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj -- --CSharpDB:DataSource=C:\data\forms.db +``` + +Run against the Fulfillment Hub sample database on an explicit local URL: + +```powershell +dotnet run --project src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj -- --urls http://127.0.0.1:5095 --CSharpDB:DataSource=C:\Users\maxim\source\Code\CSharpDB\samples\fulfillment-hub\bin\Debug\net10.0\fulfillment-hub-demo.db +``` + +After startup: + +- `/` lists available stored forms +- `/forms/orders-workbench` opens the Fulfillment Hub order runtime + +## Routes + +| Route | Purpose | +| --- | --- | +| `/` | Lists all stored forms in the configured database. | +| `/forms/{formId}` | Runs one stored form in runtime-only mode. | + +## Project Layout + +- `Program.cs` - host startup and `ICSharpDbClient` wiring +- `Configuration/FormsHostClientOptionsBuilder.cs` - configuration binding and + client option construction +- `Components/Pages/Home.razor` - root list of stored forms +- `Components/Pages/FormRuntime.razor` - runtime route wrapper around + `DataEntry` +- `wwwroot/css/app.css` - small host-specific shell styling +- `wwwroot/js/forms-host.js` - runtime pane resize interop used by `DataEntry` + +## Dependencies + +- `CSharpDB.Client` +- `CSharpDB.Admin.Forms` +- ASP.NET Core Razor Components + +## Useful Commands + +```powershell +dotnet build src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj +dotnet run --project src/CSharpDB.Admin.Forms.Web/CSharpDB.Admin.Forms.Web.csproj +``` diff --git a/src/CSharpDB.Admin.Forms.Web/_Imports.razor b/src/CSharpDB.Admin.Forms.Web/_Imports.razor new file mode 100644 index 00000000..a46b28f7 --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/_Imports.razor @@ -0,0 +1,10 @@ +@using Microsoft.AspNetCore.Components +@using Microsoft.AspNetCore.Components.Routing +@using Microsoft.AspNetCore.Components.Web +@using static Microsoft.AspNetCore.Components.Web.RenderMode +@using CSharpDB.Admin.Forms.Contracts +@using CSharpDB.Admin.Forms.Models +@using CSharpDB.Admin.Forms.Pages +@using CSharpDB.Admin.Forms.Web.Components +@using CSharpDB.Admin.Forms.Web.Components.Layout +@using CSharpDB.Client diff --git a/src/CSharpDB.Admin.Forms.Web/appsettings.json b/src/CSharpDB.Admin.Forms.Web/appsettings.json new file mode 100644 index 00000000..777e2012 --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/appsettings.json @@ -0,0 +1,12 @@ +{ + "CSharpDB": { + "Transport": "Direct", + "DataSource": "csharpdb.db", + "HostDatabase": { + "OpenMode": "HybridIncrementalDurable", + "UseWriteOptimizedPreset": true, + "HotTableNames": [], + "HotCollectionNames": [] + } + } +} diff --git a/src/CSharpDB.Admin.Forms.Web/wwwroot/css/app.css b/src/CSharpDB.Admin.Forms.Web/wwwroot/css/app.css new file mode 100644 index 00000000..6744ac48 --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/wwwroot/css/app.css @@ -0,0 +1,174 @@ +html, +body { + margin: 0; + padding: 0; + min-height: 100%; + background: #f3f5f7; + color: #16202a; + font-family: Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; +} + +body { + min-height: 100vh; +} + +a { + color: inherit; +} + +code { + font-family: "JetBrains Mono", Consolas, monospace; + font-size: 0.92em; +} + +.forms-home { + min-height: 100vh; + box-sizing: border-box; + padding: 28px 32px 40px; +} + +.forms-home__header { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 16px; + margin-bottom: 12px; +} + +.forms-home__header h1 { + margin: 0; + font-size: 28px; + font-weight: 600; + color: #101820; +} + +.forms-home__header p { + margin: 6px 0 0; + color: #51606f; + font-size: 14px; +} + +.forms-home__intro { + max-width: 880px; + margin-bottom: 20px; + color: #334454; + font-size: 14px; + line-height: 1.5; +} + +.forms-home__summary { + margin-bottom: 10px; + color: #51606f; + font-size: 13px; +} + +.forms-home__state { + padding: 16px 18px; + border: 1px solid #d4dbe3; + background: #fff; + color: #334454; + font-size: 14px; +} + +.forms-home__state--error { + border-color: #e2b3b3; + color: #8b2f2f; + background: #fff7f7; +} + +.forms-home__action { + display: inline-flex; + align-items: center; + justify-content: center; + min-height: 32px; + padding: 0 12px; + border: 1px solid #b9c6d3; + border-radius: 6px; + background: #fff; + text-decoration: none; + font-size: 13px; + font-weight: 500; + color: #16202a; +} + +.forms-home__action:hover { + background: #eef3f8; +} + +.forms-table { + width: 100%; + border-collapse: collapse; + background: #fff; + border: 1px solid #d4dbe3; +} + +.forms-table th, +.forms-table td { + padding: 12px 14px; + border-bottom: 1px solid #e6ebf1; + text-align: left; + vertical-align: middle; + font-size: 14px; +} + +.forms-table th { + font-size: 12px; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.04em; + color: #5a6876; + background: #f7f9fb; +} + +.forms-table tbody tr:hover { + background: #fafcff; +} + +.forms-table__actions { + width: 1%; + white-space: nowrap; +} + +@media (max-width: 900px) { + .forms-home { + padding: 18px 18px 28px; + } + + .forms-home__header { + flex-direction: column; + align-items: stretch; + } + + .forms-table, + .forms-table thead, + .forms-table tbody, + .forms-table tr, + .forms-table th, + .forms-table td { + display: block; + } + + .forms-table thead { + display: none; + } + + .forms-table tr { + border-bottom: 1px solid #e6ebf1; + } + + .forms-table td { + border-bottom: none; + padding: 8px 12px; + } + + .forms-table td::before { + content: attr(data-label); + display: block; + margin-bottom: 4px; + font-size: 11px; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.04em; + color: #5a6876; + } +} diff --git a/src/CSharpDB.Admin.Forms.Web/wwwroot/js/forms-host.js b/src/CSharpDB.Admin.Forms.Web/wwwroot/js/forms-host.js new file mode 100644 index 00000000..bdd77f8e --- /dev/null +++ b/src/CSharpDB.Admin.Forms.Web/wwwroot/js/forms-host.js @@ -0,0 +1,64 @@ +window.resizeInterop = { + _formEntryActive: false, + _formEntryStartX: 0, + _formEntryStartWidth: 0, + _formEntryMin: 320, + _formEntryMax: 720, + _formEntryLayout: null, + + initFormEntryPane: (layout) => { + if (!layout) return; + + const savedWidth = localStorage.getItem('csharpdb-form-entry-record-pane-width'); + if (savedWidth) { + layout.style.setProperty('--de-record-pane-width', savedWidth); + } + }, + + startFormEntryResize: (e, layout, pane, minWidth, maxWidth) => { + if (!layout || !pane) return; + + window.resizeInterop._formEntryActive = true; + window.resizeInterop._formEntryStartX = e.clientX; + window.resizeInterop._formEntryStartWidth = pane.offsetWidth; + window.resizeInterop._formEntryMin = minWidth || 320; + window.resizeInterop._formEntryMax = maxWidth || 720; + window.resizeInterop._formEntryLayout = layout; + + document.body.style.cursor = 'col-resize'; + document.body.style.userSelect = 'none'; + + document.addEventListener('mousemove', window.resizeInterop._onFormEntryMove); + document.addEventListener('mouseup', window.resizeInterop._onFormEntryUp); + }, + + _onFormEntryMove: (e) => { + if (!window.resizeInterop._formEntryActive || !window.resizeInterop._formEntryLayout) return; + + const dx = e.clientX - window.resizeInterop._formEntryStartX; + const minWidth = window.resizeInterop._formEntryMin; + const absoluteMax = Math.max(minWidth, Math.min(window.resizeInterop._formEntryMax, window.innerWidth - 360)); + let newWidth = window.resizeInterop._formEntryStartWidth - dx; + newWidth = Math.max(minWidth, Math.min(absoluteMax, newWidth)); + + window.resizeInterop._formEntryLayout.style.setProperty('--de-record-pane-width', newWidth + 'px'); + }, + + _onFormEntryUp: () => { + if (window.resizeInterop._formEntryLayout) { + const width = getComputedStyle(window.resizeInterop._formEntryLayout) + .getPropertyValue('--de-record-pane-width') + .trim(); + if (width) { + localStorage.setItem('csharpdb-form-entry-record-pane-width', width); + } + } + + window.resizeInterop._formEntryActive = false; + window.resizeInterop._formEntryLayout = null; + document.body.style.cursor = ''; + document.body.style.userSelect = ''; + document.removeEventListener('mousemove', window.resizeInterop._onFormEntryMove); + document.removeEventListener('mouseup', window.resizeInterop._onFormEntryUp); + } +}; diff --git a/src/CSharpDB.Admin.Forms/Components/Designer/PropertyInspector.razor b/src/CSharpDB.Admin.Forms/Components/Designer/PropertyInspector.razor index 96a72c3b..63540d7b 100644 --- a/src/CSharpDB.Admin.Forms/Components/Designer/PropertyInspector.razor +++ b/src/CSharpDB.Admin.Forms/Components/Designer/PropertyInspector.razor @@ -43,7 +43,7 @@
- +
@_selected.ControlId
diff --git a/src/CSharpDB.Admin.Forms/Pages/DataEntry.razor b/src/CSharpDB.Admin.Forms/Pages/DataEntry.razor index cd18662b..a3b12b07 100644 --- a/src/CSharpDB.Admin.Forms/Pages/DataEntry.razor +++ b/src/CSharpDB.Admin.Forms/Pages/DataEntry.razor @@ -9,6 +9,10 @@
+ @if (!string.IsNullOrWhiteSpace(BackHref)) + { + @BackLabel + } @(_form?.Name ?? "Loading...") @if (IsReadOnlySource) { @@ -51,7 +55,10 @@ @_validationErrors.Count error(s) } - + @if (CanOpenDesigner) + { + + }
@if (_error is not null) @@ -187,6 +194,9 @@ @code { [Parameter] public string FormId { get; set; } = string.Empty; [Parameter] public EventCallback OnOpenDesigner { get; set; } + [Parameter] public bool ShowDesignerButton { get; set; } = true; + [Parameter] public string? BackHref { get; set; } + [Parameter] public string BackLabel { get; set; } = "Forms"; private FormDefinition? _form; private FormTableDefinition? _table; @@ -223,6 +233,7 @@ private const int RecordPaneMinWidth = 320; private const int RecordPaneMaxWidth = 720; + private bool CanOpenDesigner => ShowDesignerButton && OnOpenDesigner.HasDelegate; private int TotalPages => Math.Max(1, (int)Math.Ceiling(_totalRecords / (double)_pageSize)); private int CurrentRecordOrdinal => _isNew || _recordPageIndex < 0 ? 0 : ((_page - 1) * _pageSize) + _recordPageIndex + 1; private bool HasActiveSearch => !string.IsNullOrWhiteSpace(_activeSearchColumnName) && !string.IsNullOrWhiteSpace(_activeSearchValue); diff --git a/src/CSharpDB.Admin.Forms/wwwroot/css/designer.css b/src/CSharpDB.Admin.Forms/wwwroot/css/designer.css index fb3bd869..c96e92cd 100644 --- a/src/CSharpDB.Admin.Forms/wwwroot/css/designer.css +++ b/src/CSharpDB.Admin.Forms/wwwroot/css/designer.css @@ -306,6 +306,20 @@ box-sizing: border-box; } +.pi-static-value { + width: 100%; + min-height: 30px; + padding: 6px 8px; + border: 1px solid #d0d0d0; + border-radius: 3px; + box-sizing: border-box; + display: flex; + align-items: center; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + .pi-field input[type="text"]:focus, .pi-field input[type="number"]:focus, .pi-field select:focus { @@ -424,6 +438,13 @@ font-size: 12px; } +.de-btn-link { + display: inline-flex; + align-items: center; + text-decoration: none; + color: inherit; +} + .de-btn:hover { background: #e8e8e8; } .de-btn:disabled { opacity: 0.4; cursor: default; } .de-btn-primary { background: #1a73e8; color: #fff; border-color: #1a73e8; } @@ -1784,6 +1805,21 @@ color: var(--fd-text-secondary); } +.pi-static-value { + background: var(--fd-bg-elevated); + color: var(--fd-text); + border-color: var(--fd-border); +} + +.pi-id-value { + background: color-mix(in srgb, var(--fd-bg-elevated) 88%, var(--fd-accent-soft)); + color: var(--fd-text); + font-family: var(--font-mono, "JetBrains Mono", "Fira Code", monospace); + letter-spacing: 0; + border-color: color-mix(in srgb, var(--fd-accent) 40%, var(--fd-border)); + box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--fd-accent) 14%, transparent); +} + .toolbox-item, .layer-row, .de-record-row, diff --git a/src/CSharpDB.Admin/Components/Layout/MainLayout.razor b/src/CSharpDB.Admin/Components/Layout/MainLayout.razor index 46b919f1..f7ade104 100644 --- a/src/CSharpDB.Admin/Components/Layout/MainLayout.razor +++ b/src/CSharpDB.Admin/Components/Layout/MainLayout.razor @@ -15,58 +15,64 @@
- @if (TabManager.ActiveTab is not null) + @foreach (var tab in TabManager.Tabs) { - @switch (TabManager.ActiveTab.Kind) - { - case TabKind.Welcome: - - break; - case TabKind.Query: - - break; - case TabKind.TableData: - - break; - case TabKind.ViewData: - - break; - case TabKind.Procedure: - - break; - case TabKind.Pipeline: - - break; - case TabKind.Storage: - - break; - case TabKind.TableDesigner: - - break; - case TabKind.FormDesigner: - - break; - case TabKind.FormEntry: - - break; - case TabKind.ReportDesigner: - - break; - case TabKind.ReportPreview: - - break; - } + bool isActive = TabManager.ActiveTab?.Id == tab.Id; + + }
diff --git a/src/CSharpDB.Admin/Components/Shared/DataGrid.razor b/src/CSharpDB.Admin/Components/Shared/DataGrid.razor index d0b7a0dc..781f2f96 100644 --- a/src/CSharpDB.Admin/Components/Shared/DataGrid.razor +++ b/src/CSharpDB.Admin/Components/Shared/DataGrid.razor @@ -61,10 +61,20 @@ { var colIdx = c; - +
+ + +
} @@ -193,6 +203,7 @@ private bool _sortAsc = true; private readonly Dictionary _filters = new(); private readonly Dictionary _filterInputs = new(); + private readonly Dictionary _filterModes = new(); private readonly HashSet _selectedRows = new(); private EditingCell? _editingCell; private ContextMenu? _contextMenu; @@ -244,6 +255,7 @@ _loadedAllRows = false; _filters.Clear(); _filterInputs.Clear(); + _filterModes.Clear(); Columns = null; ColumnTypes = null; PrimaryKeyColumn = null; @@ -339,11 +351,39 @@ ? activeValue : string.Empty; + private DataGridFilterMatchMode GetFilterMode(int colIdx) + => _filterModes.TryGetValue(colIdx, out var mode) + ? mode + : DataGridFilterMatchMode.Contains; + + private string GetFilterPlaceholder(int colIdx) + => GetFilterMode(colIdx) switch + { + DataGridFilterMatchMode.Exact => "Equals...", + DataGridFilterMatchMode.StartsWith => "Starts with...", + DataGridFilterMatchMode.EndsWith => "Ends with...", + _ => "Contains...", + }; + private void UpdateFilterDraft(int colIdx, string? value) { _filterInputs[colIdx] = value ?? string.Empty; } + private async Task UpdateFilterModeAsync(int colIdx, string? value) + { + if (!Enum.TryParse(value, ignoreCase: true, out var mode)) + mode = DataGridFilterMatchMode.Contains; + + if (mode == DataGridFilterMatchMode.Contains) + _filterModes.Remove(colIdx); + else + _filterModes[colIdx] = mode; + + if (_filters.TryGetValue(colIdx, out var activeValue) && !string.IsNullOrWhiteSpace(activeValue)) + await RefreshAfterFilterChangeAsync(); + } + private async Task HandleFilterKeyDown(KeyboardEventArgs e, int colIdx) { if (e.Key == "Enter") @@ -373,6 +413,19 @@ _filterInputs[colIdx] = value; } + await RefreshAfterFilterChangeAsync(); + } + + private void ResetFilterDraft(int colIdx) + { + if (_filters.TryGetValue(colIdx, out var activeValue)) + _filterInputs[colIdx] = activeValue; + else + _filterInputs.Remove(colIdx); + } + + private async Task RefreshAfterFilterChangeAsync() + { _page = 1; if (ShouldLoadAllRows()) @@ -393,14 +446,6 @@ await LoadDataAsync(); } - private void ResetFilterDraft(int colIdx) - { - if (_filters.TryGetValue(colIdx, out var activeValue)) - _filterInputs[colIdx] = activeValue; - else - _filterInputs.Remove(colIdx); - } - // ─── Row Selection ────────────── private async Task ToggleRowSelect(DataGridRow row) { @@ -619,6 +664,7 @@ continue; int colIdx = filter.Key; + var mode = GetFilterMode(colIdx); query = query.Where(row => { if (colIdx < 0 || colIdx >= row.CurrentValues.Length) @@ -631,7 +677,7 @@ byte[] bytes => $"[{bytes.Length} bytes]", _ => value.ToString() ?? string.Empty }; - return text.Contains(needle, StringComparison.OrdinalIgnoreCase); + return MatchesFilter(text, needle, mode); }); } @@ -670,6 +716,15 @@ return row.CurrentValues[colIdx]; } + private static bool MatchesFilter(string text, string needle, DataGridFilterMatchMode mode) + => mode switch + { + DataGridFilterMatchMode.Exact => string.Equals(text, needle, StringComparison.Ordinal), + DataGridFilterMatchMode.StartsWith => text.StartsWith(needle, StringComparison.OrdinalIgnoreCase), + DataGridFilterMatchMode.EndsWith => text.EndsWith(needle, StringComparison.OrdinalIgnoreCase), + _ => text.Contains(needle, StringComparison.OrdinalIgnoreCase), + }; + private sealed class CellValueComparer : IComparer { public static readonly CellValueComparer Instance = new(); @@ -911,7 +966,7 @@ string[] displayColumns = Columns ?? Array.Empty(); // Over-fetch one row so ad-hoc SQL paging can stay responsive without blocking on COUNT(*). var pageResult = await DbClient.ExecuteSqlAsync( - _queryPagingPlan.BuildPageSql(_filters, _sortCol, _sortAsc, _pageSize + 1, _page, displayColumns)); + _queryPagingPlan.BuildPageSql(_filters, _filterModes, _sortCol, _sortAsc, _pageSize + 1, _page, displayColumns)); if (pageResult.Error is not null) throw new InvalidOperationException(pageResult.Error); @@ -1045,8 +1100,7 @@ continue; string columnName = RequireIdentifier(Columns[colIdx]); - string pattern = EscapeSqlString($"%{EscapeLikePattern(filter.Value)}%"); - clauses.Add($"TEXT({columnName}) LIKE '{pattern}' ESCAPE '!'"); + clauses.Add(BuildFilterClause(colIdx, columnName, filter.Value, GetFilterMode(colIdx))); } return clauses.Count == 0 @@ -1054,6 +1108,116 @@ : $" WHERE {string.Join(" AND ", clauses)}"; } + private string BuildFilterClause(int colIdx, string columnName, string filterValue, DataGridFilterMatchMode mode) + { + string expression = $"TEXT({columnName})"; + if (mode == DataGridFilterMatchMode.Exact) + { + string? columnType = GetColumnType(colIdx); + if (TryBuildSargableExactFilterClause(columnName, columnType, filterValue, out var exactClause)) + return exactClause; + + return $"{expression} = '{EscapeSqlString(filterValue)}'"; + } + + string pattern = EscapeSqlString(BuildLikePattern(filterValue, mode)); + return $"{expression} LIKE '{pattern}' ESCAPE '!'"; + } + + private static string BuildLikePattern(string filterValue, DataGridFilterMatchMode mode) + { + string escapedFilterValue = EscapeLikePattern(filterValue); + return mode switch + { + DataGridFilterMatchMode.StartsWith => $"{escapedFilterValue}%", + DataGridFilterMatchMode.EndsWith => $"%{escapedFilterValue}", + _ => $"%{escapedFilterValue}%", + }; + } + + private string? GetColumnType(int colIdx) + => ColumnTypes is not null && colIdx >= 0 && colIdx < ColumnTypes.Length + ? ColumnTypes[colIdx] + : null; + + private static bool TryBuildSargableExactFilterClause( + string columnName, + string? columnType, + string filterValue, + out string clause) + { + clause = string.Empty; + if (string.IsNullOrWhiteSpace(columnType)) + return false; + + if (filterValue.Equals("NULL", StringComparison.OrdinalIgnoreCase)) + { + clause = IsTextColumnType(columnType) + ? $"({columnName} IS NULL OR {columnName} = 'NULL')" + : $"{columnName} IS NULL"; + return true; + } + + if (IsTextColumnType(columnType)) + { + clause = $"{columnName} = '{EscapeSqlString(filterValue)}'"; + return true; + } + + if (IsIntegerColumnType(columnType)) + { + if (long.TryParse( + filterValue, + System.Globalization.NumberStyles.Integer, + System.Globalization.CultureInfo.InvariantCulture, + out long value) && + string.Equals(filterValue, value.ToString(System.Globalization.CultureInfo.InvariantCulture), StringComparison.Ordinal)) + { + clause = $"{columnName} = {value}"; + } + else + { + clause = "0 = 1"; + } + + return true; + } + + if (IsRealColumnType(columnType)) + { + if (double.TryParse( + filterValue, + System.Globalization.NumberStyles.Float, + System.Globalization.CultureInfo.InvariantCulture, + out double value) && + double.IsFinite(value) && + string.Equals(filterValue, value.ToString(System.Globalization.CultureInfo.InvariantCulture), StringComparison.Ordinal)) + { + clause = $"{columnName} = {value.ToString(System.Globalization.CultureInfo.InvariantCulture)}"; + } + else + { + clause = "0 = 1"; + } + + return true; + } + + return false; + } + + private static bool IsTextColumnType(string columnType) + => columnType.Equals("TEXT", StringComparison.OrdinalIgnoreCase) + || columnType.Equals("VARCHAR", StringComparison.OrdinalIgnoreCase) + || columnType.Equals("CHAR", StringComparison.OrdinalIgnoreCase); + + private static bool IsIntegerColumnType(string columnType) + => columnType.Equals("INTEGER", StringComparison.OrdinalIgnoreCase) + || columnType.Equals("INT", StringComparison.OrdinalIgnoreCase); + + private static bool IsRealColumnType(string columnType) + => columnType.Equals("REAL", StringComparison.OrdinalIgnoreCase); + private string BuildOrderByClause() { if (_sortCol is not int sortCol || Columns is null || sortCol < 0 || sortCol >= Columns.Length) diff --git a/src/CSharpDB.Admin/Components/Shared/SqlEditor.razor b/src/CSharpDB.Admin/Components/Shared/SqlEditor.razor index 927b5e51..976bc553 100644 --- a/src/CSharpDB.Admin/Components/Shared/SqlEditor.razor +++ b/src/CSharpDB.Admin/Components/Shared/SqlEditor.razor @@ -1,6 +1,7 @@ @inject IJSRuntime JS +@implements IAsyncDisposable -
+
@for (int i = 1; i <= _lineCount; i++) @@ -19,6 +20,24 @@ @onscroll="SyncScroll" spellcheck="false" placeholder="@Placeholder"> + @if (_completionResult.Suggestions.Count > 0) + { +
+ @for (int i = 0; i < _completionResult.Suggestions.Count; i++) + { + var index = i; + var suggestion = _completionResult.Suggestions[i]; + + } +
+ }
@@ -27,24 +46,45 @@ [Parameter] public string Value { get; set; } = string.Empty; [Parameter] public EventCallback ValueChanged { get; set; } [Parameter] public EventCallback OnExecute { get; set; } - [Parameter] public int Height { get; set; } = 200; + [Parameter] public string Height { get; set; } = "200px"; [Parameter] public string Placeholder { get; set; } = "Enter SQL query..."; + [Parameter] public SqlCompletionCatalog CompletionCatalog { get; set; } = SqlCompletionCatalog.Empty; + [Parameter] public bool EnableCompletions { get; set; } = true; private string _highlighted = string.Empty; private int _lineCount = 1; private readonly string _editorId = $"sql-editor-{Guid.NewGuid():N}"; private readonly string _overlayId = $"sql-overlay-{Guid.NewGuid():N}"; private readonly string _lineNumId = $"sql-lines-{Guid.NewGuid():N}"; + private SqlCompletionResult _completionResult = SqlCompletionResult.Empty; + private int _selectedCompletionIndex; + private double _completionLeftPx = 12; + private double _completionTopPx = 30; + private DotNetObjectReference? _dotNetRef; protected override void OnParametersSet() { UpdateHighlight(); } + protected override async Task OnAfterRenderAsync(bool firstRender) + { + if (!firstRender) + return; + + try + { + _dotNetRef = DotNetObjectReference.Create(this); + await JS.InvokeVoidAsync("editorInterop.initSqlEditor", _editorId, _dotNetRef); + } + catch { /* circuit disconnect */ } + } + private async Task OnInput() { UpdateHighlight(); await ValueChanged.InvokeAsync(Value); + await RefreshCompletionsAsync(explicitTrigger: false); } private void UpdateHighlight() @@ -57,11 +97,126 @@ { if (e.CtrlKey && e.Key == "Enter") { + CloseCompletions(); await OnExecute.InvokeAsync(); } // Tab key support handled by browser default for textarea } + [JSInvokable] + public async Task OnSqlEditorShortcut(string action) + { + if (action == "Complete") + await RefreshCompletionsAsync(explicitTrigger: true); + } + + [JSInvokable] + public async Task OnSqlEditorCompletionKey(string key) + { + if (_completionResult.Suggestions.Count == 0) + return; + + switch (key) + { + case "ArrowDown": + _selectedCompletionIndex = (_selectedCompletionIndex + 1) % _completionResult.Suggestions.Count; + StateHasChanged(); + break; + case "ArrowUp": + _selectedCompletionIndex = (_selectedCompletionIndex - 1 + _completionResult.Suggestions.Count) % _completionResult.Suggestions.Count; + StateHasChanged(); + break; + case "Enter": + case "Tab": + await AcceptCompletionAsync(_selectedCompletionIndex); + break; + case "Escape": + CloseCompletions(); + StateHasChanged(); + break; + } + } + + private async Task RefreshCompletionsAsync(bool explicitTrigger) + { + if (!EnableCompletions) + { + CloseCompletions(); + return; + } + + EditorState state; + try + { + state = await JS.InvokeAsync("editorInterop.getEditorState", _editorId); + } + catch + { + CloseCompletions(); + return; + } + + string sql = state.Value ?? Value ?? string.Empty; + int caret = Math.Clamp(state.SelectionStart, 0, sql.Length); + var result = SqlCompletionProvider.GetCompletions(sql, caret, CompletionCatalog, explicitTrigger); + + _completionResult = result; + _selectedCompletionIndex = 0; + + if (_completionResult.Suggestions.Count > 0) + await UpdateCompletionPositionAsync(); + + StateHasChanged(); + } + + private async Task UpdateCompletionPositionAsync() + { + try + { + var coordinates = await JS.InvokeAsync("editorInterop.getCaretCoordinates", _editorId); + _completionLeftPx = Math.Max(8, coordinates.Left); + _completionTopPx = Math.Max(8, coordinates.Top); + } + catch + { + _completionLeftPx = 12; + _completionTopPx = 30; + } + } + + private async Task AcceptCompletionAsync(int index) + { + if (index < 0 || index >= _completionResult.Suggestions.Count) + return; + + var suggestion = _completionResult.Suggestions[index]; + try + { + var state = await JS.InvokeAsync( + "editorInterop.replaceEditorText", + _editorId, + suggestion.ReplacementStart, + suggestion.ReplacementEnd, + suggestion.InsertText, + suggestion.CaretPosition); + + Value = state.Value ?? string.Empty; + UpdateHighlight(); + await ValueChanged.InvokeAsync(Value); + await RefreshCompletionsAsync(explicitTrigger: false); + } + catch + { + CloseCompletions(); + } + } + + private void CloseCompletions() + { + _completionResult = SqlCompletionResult.Empty; + _selectedCompletionIndex = 0; + } + private async Task SyncScroll() { try @@ -79,4 +234,28 @@ } catch { /* circuit disconnect */ } } + + public async ValueTask DisposeAsync() + { + try + { + await JS.InvokeVoidAsync("editorInterop.disposeSqlEditor", _editorId); + } + catch { /* circuit disconnect */ } + + _dotNetRef?.Dispose(); + } + + private sealed class EditorState + { + public string? Value { get; set; } + public int SelectionStart { get; set; } + public int SelectionEnd { get; set; } + } + + private sealed class EditorCaretCoordinates + { + public double Left { get; set; } + public double Top { get; set; } + } } diff --git a/src/CSharpDB.Admin/Components/Tabs/DataTab.razor b/src/CSharpDB.Admin/Components/Tabs/DataTab.razor index 08ce36a4..6408cda8 100644 --- a/src/CSharpDB.Admin/Components/Tabs/DataTab.razor +++ b/src/CSharpDB.Admin/Components/Tabs/DataTab.razor @@ -394,6 +394,7 @@
@code { + [Parameter] public TabDescriptor? Tab { get; set; } [Parameter] public string ObjectName { get; set; } = string.Empty; [Parameter] public bool IsView { get; set; } @@ -443,19 +444,25 @@ // ─── Confirmation state ───────────── private string? _confirmDrop; + private string? _loadedObjectName; + private bool? _loadedIsView; // ─── Lifecycle ────────────────────── protected override async Task OnParametersSetAsync() { - // Check if navigation requested schema view (e.g. clicking an index/trigger in sidebar) - var activeTab = TabManager.ActiveTab; - if (activeTab?.State.TryGetValue("ShowSchema", out var ss) == true && ss is true) + if (Tab?.State.TryGetValue("ShowSchema", out var ss) == true && ss is true) { _showData = false; - activeTab.State.Remove("ShowSchema"); + Tab.State.Remove("ShowSchema"); } - await LoadSchemaAsync(); + if (!string.Equals(_loadedObjectName, ObjectName, StringComparison.Ordinal) + || _loadedIsView != IsView) + { + _loadedObjectName = ObjectName; + _loadedIsView = IsView; + await LoadSchemaAsync(); + } } private async Task LoadSchemaAsync() diff --git a/src/CSharpDB.Admin/Components/Tabs/QueryDesignerPanel.razor b/src/CSharpDB.Admin/Components/Tabs/QueryDesignerPanel.razor index 071a180c..93c43cea 100644 --- a/src/CSharpDB.Admin/Components/Tabs/QueryDesignerPanel.razor +++ b/src/CSharpDB.Admin/Components/Tabs/QueryDesignerPanel.razor @@ -338,13 +338,20 @@ } @* end grid @if *@ @* ─── SQL Preview ─── *@ -
+ 
@GeneratedSql
@* ─── Results section ─── *@ @if (_runError is not null || _lastRunSql is not null) { + @if (!_resultsCollapsed) + { +
+ }
Results @@ -408,6 +415,7 @@ // Canvas splitter private int _canvasHeightPx = 300; + private int _sqlPreviewHeightPx = 132; private string? _runError; private string? _lastRunSql; @@ -417,6 +425,7 @@ // JS interop private DotNetObjectReference? _dotNetRef; private ElementReference _canvasRef; + private ElementReference _sqlPreviewRef; // ── Lifecycle ────────────────────────────────────────────────── @@ -431,6 +440,16 @@ } catch { } } + + if (Tab?.State.TryGetValue("QueryDesignerSqlPreviewHeightPx", out var previewHeight) == true) + { + if (previewHeight is int intValue) + _sqlPreviewHeightPx = NormalizeSqlPreviewHeight(intValue); + else if (previewHeight is long longValue && longValue is >= int.MinValue and <= int.MaxValue) + _sqlPreviewHeightPx = NormalizeSqlPreviewHeight((int)longValue); + else if (previewHeight is string stringValue && int.TryParse(stringValue, out var parsed)) + _sqlPreviewHeightPx = NormalizeSqlPreviewHeight(parsed); + } } protected override async Task OnInitializedAsync() @@ -759,6 +778,35 @@ InvokeAsync(StateHasChanged); } + private async Task StartSqlPreviewResizeAsync(MouseEventArgs e) + { + try + { + await JS.InvokeVoidAsync( + "designerInterop.startPreviewSplitterDrag", + new { clientY = e.ClientY }, + _sqlPreviewRef, + _dotNetRef, + _sqlPreviewHeightPx); + } + catch + { + } + } + + [JSInvokable] + public void OnSqlPreviewHeightMoved(int newHeight) + { + _sqlPreviewHeightPx = NormalizeSqlPreviewHeight(newHeight); + if (Tab is not null) + Tab.State["QueryDesignerSqlPreviewHeightPx"] = _sqlPreviewHeightPx; + + InvokeAsync(StateHasChanged); + } + + private static int NormalizeSqlPreviewHeight(int height) + => Math.Max(96, Math.Min(320, height)); + // ── Helpers ──────────────────────────────────────────────────── private string GeneratedSql => QueryDesignerSqlBuilder.Build(_state); diff --git a/src/CSharpDB.Admin/Components/Tabs/QueryTab.razor b/src/CSharpDB.Admin/Components/Tabs/QueryTab.razor index 80d37a57..72c99ecd 100644 --- a/src/CSharpDB.Admin/Components/Tabs/QueryTab.razor +++ b/src/CSharpDB.Admin/Components/Tabs/QueryTab.razor @@ -7,10 +7,12 @@ @using CSharpDB.Client.Models @using CSharpDB.Sql @inject ICSharpDbClient DbClient +@inject IJSRuntime JS @inject DatabaseChangeService Changes @inject ToastService Toast +@implements IDisposable -
+
@if (_mode == QueryMode.Sql) { @@ -111,88 +113,100 @@
- +
+
+ +
- @if (_activeQuerySql is not null && !_loading) - { - - } - else - { -
- @if (_loading) - { -
- -
- } - else if (_error is not null) +
+ +
+ @if (_activeQuerySql is not null && !_loading) { -
- @_error -
+ } - else if (_resultColumns is not null && _resultRows is not null) + else { - - - - - @foreach (var col in _resultColumns) - { - - } - - - - @for (int r = 0; r < _resultRows.Count; r++) - { - - - @foreach (var val in _resultRows[r]) +
+ @if (_loading) + { +
+ +
+ } + else if (_error is not null) + { +
+ @_error +
+ } + else if (_resultColumns is not null && _resultRows is not null) + { +
#@col
@(r + 1)
+ + + + @foreach (var col in _resultColumns) + { + + } + + + + @for (int r = 0; r < _resultRows.Count; r++) { - + + @foreach (var val in _resultRows[r]) { - @val.ToString() + } - + } - - } - -
#@col
- @if (val is null) - { - NULL - } - else if (val is byte[] bytes) - { - [@bytes.Length bytes] - } - else +
@(r + 1) + @if (val is null) + { + NULL + } + else if (val is byte[] bytes) + { + [@bytes.Length bytes] + } + else + { + @val.ToString() + } +
- } - else if (_nonQueryMessage is not null) - { -
- @_nonQueryMessage -
- } - else - { -
- -

Run a query to see results

+ + + } + else if (_nonQueryMessage is not null) + { +
+ @_nonQueryMessage +
+ } + else + { +
+ +

Run a query to see results

+
+ }
}
- } +
} else { @@ -225,6 +239,21 @@ private const string AnalyzeSql = "ANALYZE;"; private const string TableStatsSql = "SELECT * FROM sys.table_stats ORDER BY table_name;"; private const string ColumnStatsSql = "SELECT * FROM sys.column_stats ORDER BY table_name, ordinal_position;"; + private const int DefaultEditorHeightPx = 220; + private const int MinEditorHeightPx = 120; + private const int MaxEditorHeightPx = 560; + private static readonly string[] s_systemCatalogSourceNames = + [ + "sys.tables", + "sys.columns", + "sys.indexes", + "sys.table_stats", + "sys.column_stats", + "sys.views", + "sys.triggers", + "sys.objects", + "sys.saved_queries", + ]; [Parameter] public TabDescriptor? Tab { get; set; } @@ -240,13 +269,20 @@ private string _savedQueryName = string.Empty; private string? _selectedSavedQueryName; private bool _loadingSavedQueries; + private SqlCompletionCatalog _completionCatalog = SqlCompletionCatalog.Empty; private string? _activeQuerySql; private int _queryReloadVersion; private QueryResultsStatus _queryResultsStatus = new(); + private int _editorHeightPx = DefaultEditorHeightPx; + private bool _editorHeightInitialized; + private ElementReference _queryTabRef; + private DotNetObjectReference? _dotNetRef; protected override async Task OnInitializedAsync() { + Changes.Changed += OnDatabaseObjectsChanged; await RefreshSavedQueriesAsync(); + await RefreshCompletionCatalogAsync(); } protected override void OnParametersSet() @@ -260,6 +296,40 @@ { _mode = savedMode; } + + if (TryReadTabEditorHeight(out var persistedHeight)) + _editorHeightPx = NormalizeEditorHeight(persistedHeight); + } + + protected override async Task OnAfterRenderAsync(bool firstRender) + { + if (firstRender) + _dotNetRef = DotNetObjectReference.Create(this); + + if (_editorHeightInitialized) + return; + + _editorHeightInitialized = true; + + try + { + int resolvedHeight = await JS.InvokeAsync( + "resizeInterop.initQueryEditorPane", + _queryTabRef, + _editorHeightPx, + MinEditorHeightPx, + MaxEditorHeightPx); + + resolvedHeight = NormalizeEditorHeight(resolvedHeight); + if (resolvedHeight != _editorHeightPx) + { + PersistEditorHeight(resolvedHeight); + await InvokeAsync(StateHasChanged); + } + } + catch + { + } } public async Task RunQuery() @@ -435,6 +505,63 @@ } } + private async void OnDatabaseObjectsChanged() + { + await RefreshCompletionCatalogAsync(); + await InvokeAsync(StateHasChanged); + } + + private async Task RefreshCompletionCatalogAsync() + { + try + { + var tableNamesTask = DbClient.GetTableNamesAsync(); + var viewNamesTask = DbClient.GetViewNamesAsync(); + var proceduresTask = DbClient.GetProceduresAsync(includeDisabled: false); + + await Task.WhenAll(tableNamesTask, viewNamesTask, proceduresTask); + + var sourceByName = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (string tableName in tableNamesTask.Result) + sourceByName.TryAdd(tableName, new SqlCompletionSource(tableName, SqlCompletionSourceKind.Table)); + + foreach (string viewName in viewNamesTask.Result) + sourceByName.TryAdd(viewName, new SqlCompletionSource(viewName, SqlCompletionSourceKind.View)); + + foreach (string systemCatalogName in s_systemCatalogSourceNames) + sourceByName.TryAdd(systemCatalogName, new SqlCompletionSource(systemCatalogName, SqlCompletionSourceKind.SystemCatalog)); + + var columnsBySource = new Dictionary>(StringComparer.OrdinalIgnoreCase); + foreach (string tableName in tableNamesTask.Result) + { + var schema = await DbClient.GetTableSchemaAsync(tableName); + if (schema is null) + continue; + + columnsBySource[tableName] = schema.Columns + .Select(column => new SqlCompletionColumn(column.Name, column.Type.ToString().ToUpperInvariant(), tableName)) + .ToArray(); + } + + _completionCatalog = new SqlCompletionCatalog + { + Sources = sourceByName.Values + .OrderBy(source => source.Kind) + .ThenBy(source => source.Name, StringComparer.OrdinalIgnoreCase) + .ToArray(), + ColumnsBySource = columnsBySource, + Procedures = proceduresTask.Result + .Select(procedure => procedure.Name) + .OrderBy(name => name, StringComparer.OrdinalIgnoreCase) + .ToArray(), + }; + } + catch + { + _completionCatalog = SqlCompletionCatalog.Empty; + } + } + private async Task RunExecuteCommandAsync(string procedureName, IReadOnlyDictionary args) { var result = await DbClient.ExecuteProcedureAsync(procedureName, args); @@ -511,6 +638,66 @@ _activeQuerySql = null; } + private async Task StartEditorResizeAsync(MouseEventArgs e) + { + if (_dotNetRef is null) + return; + + try + { + await JS.InvokeVoidAsync( + "resizeInterop.startQueryEditorResize", + new { clientY = e.ClientY }, + _queryTabRef, + _editorHeightPx, + MinEditorHeightPx, + MaxEditorHeightPx, + _dotNetRef); + } + catch + { + } + } + + [JSInvokable] + public Task OnQueryEditorHeightChanged(int newHeight) + { + PersistEditorHeight(newHeight); + return InvokeAsync(StateHasChanged); + } + + private void PersistEditorHeight(int height) + { + _editorHeightPx = NormalizeEditorHeight(height); + if (Tab is not null) + Tab.State["QueryEditorHeightPx"] = _editorHeightPx; + } + + private bool TryReadTabEditorHeight(out int height) + { + height = DefaultEditorHeightPx; + if (Tab?.State.TryGetValue("QueryEditorHeightPx", out var value) != true || value is null) + return false; + + switch (value) + { + case int intValue: + height = intValue; + return true; + case long longValue when longValue is >= int.MinValue and <= int.MaxValue: + height = (int)longValue; + return true; + case string stringValue when int.TryParse(stringValue, out var parsed): + height = parsed; + return true; + default: + return false; + } + } + + private static int NormalizeEditorHeight(int height) + => Math.Max(MinEditorHeightPx, Math.Min(MaxEditorHeightPx, height)); + private string GetResultSummary() { if (_activeQuerySql is not null) @@ -838,4 +1025,10 @@ error = "Invalid EXEC args: unterminated double-quoted string literal."; return false; } + + public void Dispose() + { + Changes.Changed -= OnDatabaseObjectsChanged; + _dotNetRef?.Dispose(); + } } diff --git a/src/CSharpDB.Admin/Configuration/AdminHostDatabaseOptions.cs b/src/CSharpDB.Admin/Configuration/AdminHostDatabaseOptions.cs new file mode 100644 index 00000000..f4c7f08a --- /dev/null +++ b/src/CSharpDB.Admin/Configuration/AdminHostDatabaseOptions.cs @@ -0,0 +1,166 @@ +using CSharpDB.Client; +using CSharpDB.Engine; +using Microsoft.Extensions.Configuration; + +namespace CSharpDB.Admin.Configuration; + +public enum AdminHostOpenMode +{ + HybridIncrementalDurable = 0, + Direct = 1, +} + +public sealed class AdminHostDatabaseOptions +{ + public AdminHostOpenMode OpenMode { get; init; } = AdminHostOpenMode.HybridIncrementalDurable; + + public ImplicitInsertExecutionMode ImplicitInsertExecutionMode { get; init; } = + ImplicitInsertExecutionMode.ConcurrentWriteTransactions; + + public bool UseWriteOptimizedPreset { get; init; } = true; + + public string[] HotTableNames { get; init; } = []; + + public string[] HotCollectionNames { get; init; } = []; +} + +public static class AdminClientOptionsBuilder +{ + private const string FallbackConnectionString = "Data Source=csharpdb.db"; + + public static AdminHostDatabaseOptions BindHostDatabaseOptions(IConfiguration configuration) + { + ArgumentNullException.ThrowIfNull(configuration); + + var bound = configuration.GetSection("CSharpDB:HostDatabase").Get() + ?? new AdminHostDatabaseOptions(); + + return new AdminHostDatabaseOptions + { + OpenMode = bound.OpenMode, + ImplicitInsertExecutionMode = bound.ImplicitInsertExecutionMode, + UseWriteOptimizedPreset = bound.UseWriteOptimizedPreset, + HotTableNames = bound.HotTableNames ?? [], + HotCollectionNames = bound.HotCollectionNames ?? [], + }; + } + + public static CSharpDbClientOptions Build( + IConfiguration configuration, + AdminHostDatabaseOptions hostDatabaseOptions, + CSharpDbTransport? transport, + string? endpoint) + { + ArgumentNullException.ThrowIfNull(configuration); + ArgumentNullException.ThrowIfNull(hostDatabaseOptions); + + if (!string.IsNullOrWhiteSpace(endpoint)) + { + if (transport == CSharpDbTransport.Direct || (transport is null && EndpointLooksLikeDirectPath(endpoint))) + { + return BuildDirectEndpoint(endpoint, hostDatabaseOptions, transport); + } + + return new CSharpDbClientOptions + { + Transport = transport, + Endpoint = endpoint, + }; + } + + if (transport is not null && transport != CSharpDbTransport.Direct) + { + return new CSharpDbClientOptions + { + Transport = transport, + }; + } + + return BuildDirectConnectionString( + configuration.GetConnectionString("CSharpDB") ?? FallbackConnectionString, + hostDatabaseOptions, + transport); + } + + public static CSharpDbClientOptions BuildDirectDataSource( + string dataSource, + AdminHostDatabaseOptions hostDatabaseOptions) + { + ArgumentException.ThrowIfNullOrWhiteSpace(dataSource); + ArgumentNullException.ThrowIfNull(hostDatabaseOptions); + + return new CSharpDbClientOptions + { + Transport = CSharpDbTransport.Direct, + DataSource = dataSource, + DirectDatabaseOptions = BuildDirectDatabaseOptions(hostDatabaseOptions), + HybridDatabaseOptions = BuildHybridDatabaseOptionsOrNull(hostDatabaseOptions), + }; + } + + public static DatabaseOptions BuildDirectDatabaseOptions(AdminHostDatabaseOptions hostDatabaseOptions) + { + ArgumentNullException.ThrowIfNull(hostDatabaseOptions); + + var options = new DatabaseOptions + { + ImplicitInsertExecutionMode = hostDatabaseOptions.ImplicitInsertExecutionMode, + }; + + if (!hostDatabaseOptions.UseWriteOptimizedPreset) + return options; + + return options.ConfigureStorageEngine(builder => builder.UseWriteOptimizedPreset()); + } + + public static HybridDatabaseOptions? BuildHybridDatabaseOptionsOrNull(AdminHostDatabaseOptions hostDatabaseOptions) + { + ArgumentNullException.ThrowIfNull(hostDatabaseOptions); + + if (hostDatabaseOptions.OpenMode != AdminHostOpenMode.HybridIncrementalDurable) + return null; + + return new HybridDatabaseOptions + { + PersistenceMode = HybridPersistenceMode.IncrementalDurable, + HotTableNames = hostDatabaseOptions.HotTableNames, + HotCollectionNames = hostDatabaseOptions.HotCollectionNames, + }; + } + + private static CSharpDbClientOptions BuildDirectConnectionString( + string connectionString, + AdminHostDatabaseOptions hostDatabaseOptions, + CSharpDbTransport? transport) + { + return new CSharpDbClientOptions + { + Transport = transport, + ConnectionString = connectionString, + DirectDatabaseOptions = BuildDirectDatabaseOptions(hostDatabaseOptions), + HybridDatabaseOptions = BuildHybridDatabaseOptionsOrNull(hostDatabaseOptions), + }; + } + + private static CSharpDbClientOptions BuildDirectEndpoint( + string endpoint, + AdminHostDatabaseOptions hostDatabaseOptions, + CSharpDbTransport? transport) + { + return new CSharpDbClientOptions + { + Transport = transport, + Endpoint = endpoint, + DirectDatabaseOptions = BuildDirectDatabaseOptions(hostDatabaseOptions), + HybridDatabaseOptions = BuildHybridDatabaseOptionsOrNull(hostDatabaseOptions), + }; + } + + private static bool EndpointLooksLikeDirectPath(string endpoint) + { + if (Uri.TryCreate(endpoint, UriKind.Absolute, out var uri)) + return string.Equals(uri.Scheme, Uri.UriSchemeFile, StringComparison.OrdinalIgnoreCase); + + return !endpoint.Contains("://", StringComparison.Ordinal); + } +} diff --git a/src/CSharpDB.Admin/Helpers/QueryPagingSqlBuilder.cs b/src/CSharpDB.Admin/Helpers/QueryPagingSqlBuilder.cs index 3ec6cc04..2c1cc8a7 100644 --- a/src/CSharpDB.Admin/Helpers/QueryPagingSqlBuilder.cs +++ b/src/CSharpDB.Admin/Helpers/QueryPagingSqlBuilder.cs @@ -1,4 +1,5 @@ using System.Globalization; +using CSharpDB.Admin.Models; using CSharpDB.Sql; namespace CSharpDB.Admin.Helpers; @@ -12,6 +13,8 @@ internal sealed class QueryPagingPlan private readonly QueryStatement _baseQuery; private readonly SelectStatement? _baseSelect; private readonly string _resultCteName; + private static readonly IReadOnlyDictionary EmptyFilterModes = + new Dictionary(); private QueryPagingPlan( string originalSql, @@ -77,19 +80,29 @@ public string BuildPageSql( int pageSize, int page, string[] displayColumns) + => BuildPageSql(filters, EmptyFilterModes, sortColumn, sortAscending, pageSize, page, displayColumns); + + public string BuildPageSql( + IReadOnlyDictionary filters, + IReadOnlyDictionary filterModes, + int? sortColumn, + bool sortAscending, + int pageSize, + int page, + string[] displayColumns) { ArgumentOutOfRangeException.ThrowIfLessThan(pageSize, 1); ArgumentOutOfRangeException.ThrowIfLessThan(page, 1); bool hasInteractiveTransforms = HasActiveFilters(filters) || sortColumn is not null; if (hasInteractiveTransforms - && TryBuildDirectTransforms(displayColumns, filters, sortColumn, sortAscending, out var filterExpression, out var orderBy)) + && TryBuildDirectTransforms(displayColumns, filters, filterModes, sortColumn, sortAscending, out var filterExpression, out var orderBy)) { return SerializeStatement(BuildDirectPageStatement(pageSize, page, filterExpression, orderBy)); } if (hasInteractiveTransforms) - return SerializeStatement(BuildWrappedPageStatement(pageSize, page, filters, sortColumn, sortAscending, displayColumns)); + return SerializeStatement(BuildWrappedPageStatement(pageSize, page, filters, filterModes, sortColumn, sortAscending, displayColumns)); return SerializeStatement(BuildDirectPageStatement(pageSize, page, filterExpression: null, orderBy: null)); } @@ -97,8 +110,14 @@ public string BuildPageSql( public QueryCountPlan BuildCountPlan( IReadOnlyDictionary filters, string[] displayColumns) + => BuildCountPlan(filters, EmptyFilterModes, displayColumns); + + public QueryCountPlan BuildCountPlan( + IReadOnlyDictionary filters, + IReadOnlyDictionary filterModes, + string[] displayColumns) { - if (CanUseFastCount(filters, displayColumns, out var directFilter)) + if (CanUseFastCount(filters, filterModes, displayColumns, out var directFilter)) { return new QueryCountPlan { @@ -109,7 +128,7 @@ public QueryCountPlan BuildCountPlan( return new QueryCountPlan { - Sql = SerializeStatement(BuildWrappedCountStatement(filters, displayColumns)), + Sql = SerializeStatement(BuildWrappedCountStatement(filters, filterModes, displayColumns)), ApplyBasePagination = false, }; } @@ -128,6 +147,7 @@ public int ApplyBasePaginationToCount(int rawCount) private bool CanUseFastCount( IReadOnlyDictionary filters, + IReadOnlyDictionary filterModes, string[] displayColumns, out Expression? directFilter) { @@ -144,7 +164,7 @@ private bool CanUseFastCount( if (!HasActiveFilters(filters)) return true; - return TryBuildDirectFilterExpression(displayColumns, filters, out directFilter); + return TryBuildDirectFilterExpression(displayColumns, filters, filterModes, out directFilter); } private Statement BuildFastCountStatement(Expression? directFilter) @@ -221,6 +241,7 @@ private Statement BuildWrappedPageStatement( int pageSize, int page, IReadOnlyDictionary filters, + IReadOnlyDictionary filterModes, int? sortColumn, bool sortAscending, string[] displayColumns) @@ -233,7 +254,7 @@ private Statement BuildWrappedPageStatement( IsDistinct = false, Columns = [new SelectColumn { IsStar = true }], From = new SimpleTableRef { TableName = _resultCteName }, - Where = BuildWrappedFilterExpression(filters, internalColumns), + Where = BuildWrappedFilterExpression(filters, filterModes, internalColumns), GroupBy = null, Having = null, OrderBy = BuildWrappedOrderBy(sortColumn, sortAscending, internalColumns), @@ -246,6 +267,7 @@ private Statement BuildWrappedPageStatement( private Statement BuildWrappedCountStatement( IReadOnlyDictionary filters, + IReadOnlyDictionary filterModes, string[] displayColumns) { string[]? internalColumns = HasActiveFilters(filters) ? BuildInternalColumns(displayColumns) : null; @@ -265,7 +287,7 @@ private Statement BuildWrappedCountStatement( } ], From = new SimpleTableRef { TableName = _resultCteName }, - Where = internalColumns is null ? null : BuildWrappedFilterExpression(filters, internalColumns), + Where = internalColumns is null ? null : BuildWrappedFilterExpression(filters, filterModes, internalColumns), GroupBy = null, Having = null, OrderBy = null, @@ -306,6 +328,7 @@ private Statement BuildStatement(QueryStatement query) private bool TryBuildDirectTransforms( string[] displayColumns, IReadOnlyDictionary filters, + IReadOnlyDictionary filterModes, int? sortColumn, bool sortAscending, out Expression? filterExpression, @@ -326,7 +349,7 @@ private bool TryBuildDirectTransforms( } if (HasActiveFilters(filters) - && !TryBuildDirectFilterExpression(displayColumns, filters, out filterExpression)) + && !TryBuildDirectFilterExpression(displayColumns, filters, filterModes, out filterExpression)) { return false; } @@ -351,6 +374,7 @@ private bool TryBuildDirectTransforms( private bool TryBuildDirectFilterExpression( string[] displayColumns, IReadOnlyDictionary filters, + IReadOnlyDictionary filterModes, out Expression? filterExpression) { filterExpression = null; @@ -363,7 +387,7 @@ private bool TryBuildDirectFilterExpression( if (!TryGetDirectColumnExpression(displayColumns, filter.Key, out var sourceExpression)) return false; - var predicate = BuildContainsPredicate(sourceExpression, filter.Value); + var predicate = BuildFilterPredicate(sourceExpression, filter.Value, GetFilterMode(filterModes, filter.Key)); filterExpression = filterExpression is null ? predicate : new BinaryExpression @@ -406,28 +430,43 @@ private bool TryGetDirectColumnExpression( return true; } - private static Expression BuildContainsPredicate(Expression sourceExpression, string filterValue) - { - return new LikeExpression - { - Operand = new FunctionCallExpression - { - FunctionName = "TEXT", - Arguments = [sourceExpression], - IsStarArg = false, - }, - Pattern = new LiteralExpression + private static Expression BuildFilterPredicate( + Expression sourceExpression, + string filterValue, + DataGridFilterMatchMode mode) + => mode == DataGridFilterMatchMode.Exact + ? new BinaryExpression { - Value = $"%{EscapeLikePattern(filterValue)}%", - LiteralType = TokenType.StringLiteral, - }, - EscapeChar = new LiteralExpression + Op = BinaryOp.Equals, + Left = BuildTextExpression(sourceExpression), + Right = new LiteralExpression + { + Value = filterValue, + LiteralType = TokenType.StringLiteral, + }, + } + : new LikeExpression { - Value = "!", - LiteralType = TokenType.StringLiteral, - }, + Operand = BuildTextExpression(sourceExpression), + Pattern = new LiteralExpression + { + Value = BuildLikePattern(filterValue, mode), + LiteralType = TokenType.StringLiteral, + }, + EscapeChar = new LiteralExpression + { + Value = "!", + LiteralType = TokenType.StringLiteral, + }, + }; + + private static FunctionCallExpression BuildTextExpression(Expression sourceExpression) + => new() + { + FunctionName = "TEXT", + Arguments = [sourceExpression], + IsStarArg = false, }; - } private static Expression? CombineAnd(Expression? left, Expression? right) { @@ -469,6 +508,7 @@ private static string[] BuildInternalColumns(string[] displayColumns) private static Expression? BuildWrappedFilterExpression( IReadOnlyDictionary filters, + IReadOnlyDictionary filterModes, string[] internalColumns) { Expression? filterExpression = null; @@ -481,9 +521,10 @@ private static string[] BuildInternalColumns(string[] displayColumns) if (filter.Key < 0 || filter.Key >= internalColumns.Length) continue; - var predicate = BuildContainsPredicate( + var predicate = BuildFilterPredicate( new ColumnRefExpression { ColumnName = internalColumns[filter.Key] }, - filter.Value); + filter.Value, + GetFilterMode(filterModes, filter.Key)); filterExpression = filterExpression is null ? predicate @@ -566,12 +607,30 @@ private static bool IsAggregateFunction(string name) private static bool HasActiveFilters(IReadOnlyDictionary filters) => filters.Values.Any(value => !string.IsNullOrWhiteSpace(value)); + private static DataGridFilterMatchMode GetFilterMode( + IReadOnlyDictionary filterModes, + int columnIndex) + => filterModes.TryGetValue(columnIndex, out var mode) + ? mode + : DataGridFilterMatchMode.Contains; + private static string EscapeLikePattern(string value) => value .Replace("!", "!!", StringComparison.Ordinal) .Replace("%", "!%", StringComparison.Ordinal) .Replace("_", "!_", StringComparison.Ordinal); + private static string BuildLikePattern(string filterValue, DataGridFilterMatchMode mode) + { + string escapedFilterValue = EscapeLikePattern(filterValue); + return mode switch + { + DataGridFilterMatchMode.StartsWith => $"{escapedFilterValue}%", + DataGridFilterMatchMode.EndsWith => $"%{escapedFilterValue}", + _ => $"%{escapedFilterValue}%", + }; + } + private static string BuildUniqueResultCteName(IEnumerable ctes) { var existingNames = new HashSet( diff --git a/src/CSharpDB.Admin/Models/DataGridFilterMatchMode.cs b/src/CSharpDB.Admin/Models/DataGridFilterMatchMode.cs new file mode 100644 index 00000000..5f3dd5b5 --- /dev/null +++ b/src/CSharpDB.Admin/Models/DataGridFilterMatchMode.cs @@ -0,0 +1,9 @@ +namespace CSharpDB.Admin.Models; + +public enum DataGridFilterMatchMode +{ + Contains = 0, + Exact = 1, + StartsWith = 2, + EndsWith = 3, +} diff --git a/src/CSharpDB.Admin/Program.cs b/src/CSharpDB.Admin/Program.cs index 4d424fc4..0adde517 100644 --- a/src/CSharpDB.Admin/Program.cs +++ b/src/CSharpDB.Admin/Program.cs @@ -1,3 +1,4 @@ +using CSharpDB.Admin.Configuration; using CSharpDB.Admin.Components; using CSharpDB.Admin.Forms.Services; using CSharpDB.Admin.Reports.Services; @@ -9,32 +10,22 @@ builder.Services.AddRazorComponents() .AddInteractiveServerComponents(); +builder.Services.AddSingleton(sp => + AdminClientOptionsBuilder.BindHostDatabaseOptions(sp.GetRequiredService())); builder.Services.AddSingleton(sp => { var configuration = sp.GetRequiredService(); + var hostDatabaseOptions = sp.GetRequiredService(); string? endpoint = configuration["CSharpDB:Endpoint"]; CSharpDbTransport? transport = ParseTransport(configuration["CSharpDB:Transport"]); - CSharpDbClientOptions options; - if (!string.IsNullOrWhiteSpace(endpoint)) - { - options = new CSharpDbClientOptions - { - Transport = transport, - Endpoint = endpoint, - }; - } - else - { - options = new CSharpDbClientOptions - { - Transport = transport, - ConnectionString = configuration.GetConnectionString("CSharpDB") - ?? "Data Source=csharpdb.db", - }; - } + CSharpDbClientOptions options = AdminClientOptionsBuilder.Build( + configuration, + hostDatabaseOptions, + transport, + endpoint); - return new DatabaseClientHolder(CSharpDbClient.Create(options)); + return new DatabaseClientHolder(CSharpDbClient.Create(options), hostDatabaseOptions); }); builder.Services.AddSingleton(sp => sp.GetRequiredService()); builder.Services.AddScoped(); @@ -47,7 +38,7 @@ var app = builder.Build(); -// Open the database connection at startup (before any requests arrive) +// Warm the in-process database instance before any requests arrive. await using (var scope = app.Services.CreateAsyncScope()) { var dbClient = scope.ServiceProvider.GetRequiredService(); diff --git a/src/CSharpDB.Admin/README.md b/src/CSharpDB.Admin/README.md index 6365195b..2d72b59a 100644 --- a/src/CSharpDB.Admin/README.md +++ b/src/CSharpDB.Admin/README.md @@ -10,9 +10,10 @@ database objects. - object explorer for user tables, system tables, forms, reports, views, triggers, saved queries, and procedures -- table browsing with insert, update, delete, and schema views +- table browsing with insert, update, delete, per-column `LIKE` placement and `=` filters, + and schema views - table designer for creating tables -- SQL query tabs with paged results +- SQL query tabs with paged results and guided SQL completions - procedure editor and execution surface - storage inspection and maintenance views - ETL pipeline designer, JSON package editor, stored pipeline catalog, and run @@ -26,7 +27,9 @@ The host registers one `ICSharpDbClient` through `DatabaseClientHolder`. Configuration supports two shapes: -- direct embedded database access through `ConnectionStrings:CSharpDB` +- direct embedded database access through `ConnectionStrings:CSharpDB`; local + direct mode keeps a warm in-process database instance and opens it with + hybrid incremental-durable options by default - remote access through `CSharpDB:Transport` plus `CSharpDB:Endpoint` Default `appsettings.json` uses direct mode: @@ -34,7 +37,14 @@ Default `appsettings.json` uses direct mode: ```json { "CSharpDB": { - "Transport": "direct" + "Transport": "direct", + "HostDatabase": { + "OpenMode": "HybridIncrementalDurable", + "ImplicitInsertExecutionMode": "ConcurrentWriteTransactions", + "UseWriteOptimizedPreset": true, + "HotTableNames": [], + "HotCollectionNames": [] + } }, "ConnectionStrings": { "CSharpDB": "Data Source=relational.db" @@ -42,9 +52,16 @@ Default `appsettings.json` uses direct mode: } ``` +In this Admin context, hybrid means a warm in-process database instance that +stays alive for the Admin app lifetime. It is still durable and persists to the +configured data source; it is not an in-memory-only mode. Set +`CSharpDB:HostDatabase:OpenMode` to `Direct` to use the plain direct open path +instead of `Database.OpenHybridAsync`. + The app opens the configured database during startup by calling `ICSharpDbClient.GetInfoAsync()`, so invalid configuration fails before the UI -accepts requests. +accepts requests. The same database instance is reused until the Admin app +shuts down or the user switches to a different database. ## Running Locally @@ -59,11 +76,20 @@ The development launch profile uses: ## Configuration Examples -Direct file-backed database: +Direct local warm database: + +```powershell +$env:ConnectionStrings__CSharpDB = "Data Source=C:\data\app.db" +$env:CSharpDB__Transport = "direct" +dotnet run --project src/CSharpDB.Admin/CSharpDB.Admin.csproj +``` + +Plain direct file-open opt-out: ```powershell $env:ConnectionStrings__CSharpDB = "Data Source=C:\data\app.db" $env:CSharpDB__Transport = "direct" +$env:CSharpDB__HostDatabase__OpenMode = "Direct" dotnet run --project src/CSharpDB.Admin/CSharpDB.Admin.csproj ``` diff --git a/src/CSharpDB.Admin/Services/DatabaseClientHolder.cs b/src/CSharpDB.Admin/Services/DatabaseClientHolder.cs index 33dd083f..3c3652c3 100644 --- a/src/CSharpDB.Admin/Services/DatabaseClientHolder.cs +++ b/src/CSharpDB.Admin/Services/DatabaseClientHolder.cs @@ -1,4 +1,5 @@ using System.Text.Json; +using CSharpDB.Admin.Configuration; using CSharpDB.Client; using CSharpDB.Client.Models; using CSharpDB.Storage.Diagnostics; @@ -13,22 +14,21 @@ namespace CSharpDB.Admin.Services; public sealed class DatabaseClientHolder : ICSharpDbClient { private ICSharpDbClient _inner; + private readonly AdminHostDatabaseOptions _hostDatabaseOptions; private readonly object _lock = new(); public event Action? DatabaseChanged; - public DatabaseClientHolder(ICSharpDbClient initial) + public DatabaseClientHolder(ICSharpDbClient initial, AdminHostDatabaseOptions hostDatabaseOptions) { _inner = initial; + _hostDatabaseOptions = hostDatabaseOptions; } public async Task SwitchAsync(string databasePath) { - var newClient = CSharpDbClient.Create(new CSharpDbClientOptions - { - Transport = CSharpDbTransport.Direct, - DataSource = databasePath, - }); + var newClient = CSharpDbClient.Create( + AdminClientOptionsBuilder.BuildDirectDataSource(databasePath, _hostDatabaseOptions)); // Verify the new database is accessible before swapping. await newClient.GetInfoAsync(); diff --git a/src/CSharpDB.Admin/appsettings.json b/src/CSharpDB.Admin/appsettings.json index 5c41f5a5..d76efd9d 100644 --- a/src/CSharpDB.Admin/appsettings.json +++ b/src/CSharpDB.Admin/appsettings.json @@ -1,9 +1,16 @@ { "CSharpDB": { - "Transport": "direct" + "Transport": "direct", + "HostDatabase": { + "OpenMode": "HybridIncrementalDurable", + "ImplicitInsertExecutionMode": "ConcurrentWriteTransactions", + "UseWriteOptimizedPreset": true, + "HotTableNames": [], + "HotCollectionNames": [] + } }, "ConnectionStrings": { - "CSharpDB": "Data Source=relational.db" + "CSharpDB": "Data Source=fulfillment-hub-demo" }, "Logging": { "LogLevel": { diff --git a/src/CSharpDB.Admin/wwwroot/css/app.css b/src/CSharpDB.Admin/wwwroot/css/app.css index 784aaf02..558035a6 100644 --- a/src/CSharpDB.Admin/wwwroot/css/app.css +++ b/src/CSharpDB.Admin/wwwroot/css/app.css @@ -161,6 +161,12 @@ body { .content-area { flex: 1; + overflow: hidden; + position: relative; +} + +.tab-content-panel { + height: 100%; overflow: auto; } @@ -704,6 +710,60 @@ body { .sql-editor-area textarea.sql-editor::placeholder { color: var(--text-muted); } +.sql-completion-popup { + position: absolute; + z-index: 20; + width: min(360px, calc(100% - 24px)); + max-height: 236px; + overflow-y: auto; + padding: 4px; + border: 1px solid var(--border-color); + border-radius: var(--radius-sm); + background: var(--bg-elevated); + box-shadow: 0 10px 26px rgba(0,0,0,0.28); +} + +.sql-completion-item { + display: grid; + grid-template-columns: minmax(0, 1fr) auto; + align-items: center; + gap: 10px; + width: 100%; + min-height: 28px; + padding: 5px 8px; + border: none; + border-radius: var(--radius-sm); + background: transparent; + color: var(--text-secondary); + font-family: var(--font-mono); + font-size: 12px; + text-align: left; + cursor: pointer; +} + +.sql-completion-item:hover, +.sql-completion-item.active { + background: var(--bg-active); + color: var(--text-primary); +} + +.sql-completion-label { + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; + font-weight: 600; +} + +.sql-completion-detail { + overflow: hidden; + max-width: 150px; + color: var(--text-muted); + font-family: var(--font-ui); + font-size: 11px; + text-overflow: ellipsis; + white-space: nowrap; +} + /* Syntax Highlighting */ .hl-keyword { color: var(--hl-keyword); font-weight: 600; } .hl-function { color: var(--hl-function); } @@ -839,8 +899,29 @@ body { border-bottom: 2px solid var(--border-color); } +.filter-control { + display: flex; + align-items: center; + gap: 4px; + min-width: 150px; +} + +.filter-mode-select { + width: 54px; + flex: 0 0 54px; + padding: 3px 4px; + border: 1px solid var(--border-color); + border-radius: var(--radius-sm); + background: var(--bg-elevated); + color: var(--text-secondary); + font-size: 11px; + font-family: var(--font-ui); + outline: none; +} + .filter-input { width: 100%; + min-width: 0; padding: 3px 6px; border: 1px solid var(--border-color); border-radius: var(--radius-sm); @@ -851,7 +932,8 @@ body { outline: none; } -.filter-input:focus { border-color: var(--accent-blue); } +.filter-input:focus, +.filter-mode-select:focus { border-color: var(--accent-blue); } /* Inline editing */ .editing { @@ -1353,6 +1435,74 @@ body { display: flex; flex-direction: column; height: 100%; + min-height: 0; +} + +.query-workspace { + display: flex; + flex: 1; + flex-direction: column; + min-height: 0; + overflow: hidden; +} + +.query-editor-panel { + flex: 0 0 var(--query-editor-height, 220px); + min-height: 0; + overflow: hidden; +} + +.query-editor-panel .sql-editor-container { + height: 100%; + border-bottom: none; +} + +.query-editor-splitter { + position: relative; + height: 12px; + flex-shrink: 0; + background: var(--bg-secondary); + border-top: 1px solid var(--border-color); + border-bottom: 1px solid var(--border-color); + cursor: row-resize; + transition: background var(--transition-fast), border-color var(--transition-fast); +} + +.query-editor-splitter:hover { + background: var(--bg-active); + border-color: color-mix(in srgb, var(--accent-blue) 35%, var(--border-color)); +} + +.query-editor-splitter::after { + content: ""; + position: absolute; + top: 50%; + left: 50%; + width: 68px; + height: 4px; + transform: translate(-50%, -50%); + border-radius: 999px; + background: color-mix(in srgb, var(--text-muted) 78%, transparent); + box-shadow: 0 0 0 1px color-mix(in srgb, var(--bg-primary) 55%, transparent); + transition: background var(--transition-fast), width var(--transition-fast); +} + +.query-editor-splitter:hover::after { + width: 84px; + background: color-mix(in srgb, var(--accent-blue) 80%, white 8%); +} + +.query-results-panel { + display: flex; + flex: 1; + flex-direction: column; + min-height: 120px; + overflow: hidden; +} + +.query-results-panel > .grid-container { + flex: 1; + min-height: 0; } .query-saved-bar { @@ -2740,7 +2890,7 @@ body { padding: 10px 14px; border-top: 1px solid var(--border-color); background: var(--bg-secondary); - max-height: 120px; + min-height: 96px; overflow-y: auto; } .designer-sql-preview pre { @@ -2753,6 +2903,41 @@ body { word-break: break-all; } +.designer-preview-splitter { + position: relative; + height: 12px; + flex-shrink: 0; + background: var(--bg-secondary); + border-top: 1px solid var(--border-color); + border-bottom: 1px solid var(--border-color); + cursor: row-resize; + transition: background var(--transition-fast), border-color var(--transition-fast); +} + +.designer-preview-splitter:hover { + background: var(--bg-active); + border-color: color-mix(in srgb, var(--accent-blue) 35%, var(--border-color)); +} + +.designer-preview-splitter::after { + content: ""; + position: absolute; + top: 50%; + left: 50%; + width: 68px; + height: 4px; + transform: translate(-50%, -50%); + border-radius: 999px; + background: color-mix(in srgb, var(--text-muted) 78%, transparent); + box-shadow: 0 0 0 1px color-mix(in srgb, var(--bg-primary) 55%, transparent); + transition: background var(--transition-fast), width var(--transition-fast); +} + +.designer-preview-splitter:hover::after { + width: 84px; + background: color-mix(in srgb, var(--accent-blue) 80%, white 8%); +} + /* Edit join inline popup */ .designer-join-popup { display: flex; diff --git a/src/CSharpDB.Admin/wwwroot/js/interop.js b/src/CSharpDB.Admin/wwwroot/js/interop.js index c24f7c0b..43fa7b20 100644 --- a/src/CSharpDB.Admin/wwwroot/js/interop.js +++ b/src/CSharpDB.Admin/wwwroot/js/interop.js @@ -67,6 +67,13 @@ window.resizeInterop = { _formEntryMin: 320, _formEntryMax: 720, _formEntryLayout: null, + _queryEditorActive: false, + _queryEditorStartY: 0, + _queryEditorStartHeight: 0, + _queryEditorMin: 120, + _queryEditorMax: 560, + _queryEditorLayout: null, + _queryEditorDotNetRef: null, initSidebar: (dotNetRef, minWidth, maxWidth) => { window.resizeInterop._dotNetRef = dotNetRef; @@ -158,6 +165,77 @@ window.resizeInterop = { document.body.style.userSelect = ''; document.removeEventListener('mousemove', window.resizeInterop._onFormEntryMove); document.removeEventListener('mouseup', window.resizeInterop._onFormEntryUp); + }, + + initQueryEditorPane: (layout, fallbackHeight, minHeight, maxHeight) => { + if (!layout) return fallbackHeight || 220; + + const min = minHeight || 120; + const configuredMax = maxHeight || 560; + const layoutMax = Math.max(min, layout.clientHeight - 140); + const max = Math.max(min, Math.min(configuredMax, layoutMax)); + + const storedHeight = parseInt(localStorage.getItem('csharpdb-query-editor-height') || '', 10); + const baseHeight = Number.isFinite(storedHeight) ? storedHeight : (fallbackHeight || 220); + const resolvedHeight = Math.max(min, Math.min(max, baseHeight)); + + layout.style.setProperty('--query-editor-height', resolvedHeight + 'px'); + return resolvedHeight; + }, + + startQueryEditorResize: (e, layout, currentHeight, minHeight, maxHeight, dotNetRef) => { + if (!layout) return; + + window.resizeInterop._queryEditorActive = true; + window.resizeInterop._queryEditorStartY = e.clientY; + window.resizeInterop._queryEditorStartHeight = currentHeight || 220; + window.resizeInterop._queryEditorMin = minHeight || 120; + window.resizeInterop._queryEditorMax = maxHeight || 560; + window.resizeInterop._queryEditorLayout = layout; + window.resizeInterop._queryEditorDotNetRef = dotNetRef || null; + + document.body.style.cursor = 'row-resize'; + document.body.style.userSelect = 'none'; + + document.addEventListener('mousemove', window.resizeInterop._onQueryEditorMove); + document.addEventListener('mouseup', window.resizeInterop._onQueryEditorUp); + }, + + _onQueryEditorMove: (e) => { + if (!window.resizeInterop._queryEditorActive || !window.resizeInterop._queryEditorLayout) return; + + const dy = e.clientY - window.resizeInterop._queryEditorStartY; + const min = window.resizeInterop._queryEditorMin; + const configuredMax = window.resizeInterop._queryEditorMax; + const layoutMax = Math.max(min, window.resizeInterop._queryEditorLayout.clientHeight - 140); + const max = Math.max(min, Math.min(configuredMax, layoutMax)); + let nextHeight = window.resizeInterop._queryEditorStartHeight + dy; + nextHeight = Math.max(min, Math.min(max, nextHeight)); + + window.resizeInterop._queryEditorLayout.style.setProperty('--query-editor-height', nextHeight + 'px'); + }, + + _onQueryEditorUp: () => { + const layout = window.resizeInterop._queryEditorLayout; + const dotNetRef = window.resizeInterop._queryEditorDotNetRef; + + if (layout) { + const value = parseInt(getComputedStyle(layout).getPropertyValue('--query-editor-height') || '', 10); + if (Number.isFinite(value)) { + localStorage.setItem('csharpdb-query-editor-height', value + 'px'); + if (dotNetRef) { + dotNetRef.invokeMethodAsync('OnQueryEditorHeightChanged', value); + } + } + } + + window.resizeInterop._queryEditorActive = false; + window.resizeInterop._queryEditorLayout = null; + window.resizeInterop._queryEditorDotNetRef = null; + document.body.style.cursor = ''; + document.body.style.userSelect = ''; + document.removeEventListener('mousemove', window.resizeInterop._onQueryEditorMove); + document.removeEventListener('mouseup', window.resizeInterop._onQueryEditorUp); } }; @@ -236,6 +314,34 @@ window.designerInterop = { dotNetRef.invokeMethodAsync('OnSplitterMoved', Math.round(finalH)); }; + document.body.style.cursor = 'row-resize'; + document.body.style.userSelect = 'none'; + document.addEventListener('mousemove', onMove); + document.addEventListener('mouseup', onUp); + }, + + startPreviewSplitterDrag: (e, previewElement, dotNetRef, currentHeight) => { + const preview = previewElement; + if (!preview) return; + + const startY = e.clientY; + const startH = currentHeight || preview.offsetHeight || 132; + + const onMove = (ev) => { + const dy = ev.clientY - startY; + const newH = Math.max(96, Math.min(320, startH + dy)); + preview.style.height = newH + 'px'; + }; + + const onUp = () => { + document.removeEventListener('mousemove', onMove); + document.removeEventListener('mouseup', onUp); + document.body.style.cursor = ''; + document.body.style.userSelect = ''; + const finalH = parseFloat(preview.style.height) || startH; + dotNetRef.invokeMethodAsync('OnSqlPreviewHeightMoved', Math.round(finalH)); + }; + document.body.style.cursor = 'row-resize'; document.body.style.userSelect = 'none'; document.addEventListener('mousemove', onMove); @@ -245,11 +351,129 @@ window.designerInterop = { // SQL editor scroll sync window.editorInterop = { + _editors: new Map(), + + initSqlEditor: (editorId, dotNetRef) => { + const textarea = document.getElementById(editorId); + if (!textarea) return; + + window.editorInterop.disposeSqlEditor(editorId); + + const keydown = (e) => { + if (e.ctrlKey && e.key === ' ') { + e.preventDefault(); + dotNetRef.invokeMethodAsync('OnSqlEditorShortcut', 'Complete'); + return; + } + + const completionOpen = !!textarea + .closest('.sql-editor-area') + ?.querySelector('.sql-completion-popup'); + + if (!completionOpen) return; + + if (e.key === 'ArrowDown' || e.key === 'ArrowUp' || e.key === 'Enter' || e.key === 'Tab' || e.key === 'Escape') { + e.preventDefault(); + dotNetRef.invokeMethodAsync('OnSqlEditorCompletionKey', e.key); + } + }; + + textarea.addEventListener('keydown', keydown); + window.editorInterop._editors.set(editorId, { keydown }); + }, + + disposeSqlEditor: (editorId) => { + const registration = window.editorInterop._editors.get(editorId); + if (!registration) return; + + const textarea = document.getElementById(editorId); + if (textarea) { + textarea.removeEventListener('keydown', registration.keydown); + } + + window.editorInterop._editors.delete(editorId); + }, + + getEditorState: (editorId) => { + const textarea = document.getElementById(editorId); + if (!textarea) return { value: '', selectionStart: 0, selectionEnd: 0 }; + + return { + value: textarea.value, + selectionStart: textarea.selectionStart || 0, + selectionEnd: textarea.selectionEnd || 0 + }; + }, + + replaceEditorText: (editorId, start, end, insertText, caretPosition) => { + const textarea = document.getElementById(editorId); + if (!textarea) return { value: '', selectionStart: 0, selectionEnd: 0 }; + + const value = textarea.value || ''; + const safeStart = Math.max(0, Math.min(start || 0, value.length)); + const safeEnd = Math.max(safeStart, Math.min(end || safeStart, value.length)); + const nextValue = value.slice(0, safeStart) + (insertText || '') + value.slice(safeEnd); + const nextCaret = Math.max(0, Math.min(caretPosition ?? (safeStart + (insertText || '').length), nextValue.length)); + + textarea.value = nextValue; + textarea.focus(); + textarea.setSelectionRange(nextCaret, nextCaret); + textarea.dispatchEvent(new Event('input', { bubbles: true })); + + return { + value: nextValue, + selectionStart: nextCaret, + selectionEnd: nextCaret + }; + }, + + getCaretCoordinates: (editorId) => { + const textarea = document.getElementById(editorId); + if (!textarea) return { left: 12, top: 30 }; + + const style = window.getComputedStyle(textarea); + const mirror = document.createElement('div'); + mirror.style.position = 'absolute'; + mirror.style.visibility = 'hidden'; + mirror.style.whiteSpace = 'pre-wrap'; + mirror.style.wordWrap = 'break-word'; + mirror.style.overflowWrap = 'break-word'; + mirror.style.boxSizing = style.boxSizing; + mirror.style.width = textarea.clientWidth + 'px'; + mirror.style.font = style.font; + mirror.style.fontFamily = style.fontFamily; + mirror.style.fontSize = style.fontSize; + mirror.style.fontWeight = style.fontWeight; + mirror.style.lineHeight = style.lineHeight; + mirror.style.letterSpacing = style.letterSpacing; + mirror.style.padding = style.padding; + mirror.style.border = style.border; + mirror.style.left = '-9999px'; + mirror.style.top = '0'; + + const before = (textarea.value || '').slice(0, textarea.selectionStart || 0); + mirror.appendChild(document.createTextNode(before)); + const marker = document.createElement('span'); + marker.textContent = '\u200b'; + mirror.appendChild(marker); + document.body.appendChild(mirror); + + const lineHeight = parseFloat(style.lineHeight) || 19.2; + const area = textarea.closest('.sql-editor-area'); + const areaWidth = area?.clientWidth || textarea.clientWidth; + const left = Math.min(Math.max(marker.offsetLeft - textarea.scrollLeft, 8), Math.max(8, areaWidth - 280)); + const top = marker.offsetTop - textarea.scrollTop + lineHeight + 4; + + document.body.removeChild(mirror); + + return { left, top }; + }, + syncScroll: (editorId) => { const textarea = document.getElementById(editorId); if (!textarea) return; const overlay = textarea.previousElementSibling; - const lineNums = textarea.parentElement?.querySelector('.sql-line-numbers'); + const lineNums = textarea.closest('.sql-editor-wrapper')?.querySelector('.sql-line-numbers'); if (overlay) { overlay.scrollTop = textarea.scrollTop; overlay.scrollLeft = textarea.scrollLeft; diff --git a/src/CSharpDB.Api/CSharpDB.Api.csproj b/src/CSharpDB.Api/CSharpDB.Api.csproj index 0211cc9d..844bc9a0 100644 --- a/src/CSharpDB.Api/CSharpDB.Api.csproj +++ b/src/CSharpDB.Api/CSharpDB.Api.csproj @@ -9,8 +9,8 @@ - - + + diff --git a/src/CSharpDB.Api/CSharpDbRestApiHostExtensions.cs b/src/CSharpDB.Api/CSharpDbRestApiHostExtensions.cs new file mode 100644 index 00000000..dbc64084 --- /dev/null +++ b/src/CSharpDB.Api/CSharpDbRestApiHostExtensions.cs @@ -0,0 +1,118 @@ +using System.Text.Json; +using System.Text.Json.Serialization; +using CSharpDB.Api.Endpoints; +using CSharpDB.Api.Middleware; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.DependencyInjection; +using Scalar.AspNetCore; + +namespace CSharpDB.Api; + +public sealed class CSharpDbRestApiHostOptions +{ + public string RoutePrefix { get; set; } = "/api"; + + public string OpenApiTitle { get; set; } = "CSharpDB API"; + + public bool MapDevelopmentOpenApi { get; set; } = true; + + public bool ApplyMiddlewareToApiOnly { get; set; } +} + +public static class CSharpDbRestApiHostExtensions +{ + public static IServiceCollection AddCSharpDbRestApi(this IServiceCollection services) + { + services.AddOpenApi(); + + services.AddCors(options => + { + options.AddDefaultPolicy(policy => + { + policy.AllowAnyOrigin() + .AllowAnyMethod() + .AllowAnyHeader(); + }); + }); + + services.ConfigureHttpJsonOptions(options => + { + options.SerializerOptions.PropertyNamingPolicy = JsonNamingPolicy.CamelCase; + options.SerializerOptions.DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull; + options.SerializerOptions.Converters.Add(new JsonStringEnumConverter(JsonNamingPolicy.CamelCase)); + }); + + return services; + } + + public static WebApplication MapCSharpDbRestApi(this WebApplication app) + => app.MapCSharpDbRestApi(configure: null); + + public static WebApplication MapCSharpDbRestApi( + this WebApplication app, + Action? configure) + { + var options = new CSharpDbRestApiHostOptions(); + configure?.Invoke(options); + + string routePrefix = NormalizeRoutePrefix(options.RoutePrefix); + var apiPath = new PathString(routePrefix); + + if (options.ApplyMiddlewareToApiOnly) + { + app.UseWhen( + context => context.Request.Path.StartsWithSegments(apiPath), + branch => + { + branch.UseCors(); + branch.UseMiddleware(); + }); + } + else + { + app.UseCors(); + app.UseMiddleware(); + } + + if (options.MapDevelopmentOpenApi && app.Environment.IsDevelopment()) + { + app.MapOpenApi(); + app.MapScalarApiReference(scalar => + { + scalar.WithTitle(options.OpenApiTitle); + scalar.WithDefaultHttpClient(ScalarTarget.CSharp, ScalarClient.HttpClient); + }); + } + + var api = app.MapGroup(routePrefix); + + api.MapTableEndpoints(); + api.MapRowEndpoints(); + api.MapIndexEndpoints(); + api.MapViewEndpoints(); + api.MapTriggerEndpoints(); + api.MapProcedureEndpoints(); + api.MapSavedQueryEndpoints(); + api.MapSqlEndpoints(); + api.MapPipelineEndpoints(); + api.MapTransactionEndpoints(); + api.MapCollectionEndpoints(); + api.MapSchemaEndpoints(); + api.MapInspectEndpoints(); + api.MapMaintenanceEndpoints(); + + return app; + } + + private static string NormalizeRoutePrefix(string routePrefix) + { + if (string.IsNullOrWhiteSpace(routePrefix)) + return "/api"; + + routePrefix = routePrefix.Trim(); + return routePrefix.StartsWith("/", StringComparison.Ordinal) + ? routePrefix + : "/" + routePrefix; + } +} diff --git a/src/CSharpDB.Api/Program.cs b/src/CSharpDB.Api/Program.cs index fb6b8129..ac677e3e 100644 --- a/src/CSharpDB.Api/Program.cs +++ b/src/CSharpDB.Api/Program.cs @@ -1,9 +1,5 @@ -using System.Text.Json; -using System.Text.Json.Serialization; +using CSharpDB.Api; using CSharpDB.Client; -using CSharpDB.Api.Endpoints; -using CSharpDB.Api.Middleware; -using Scalar.AspNetCore; var builder = WebApplication.CreateBuilder(args); @@ -15,24 +11,7 @@ ?? "Data Source=csharpdb.db", }); -builder.Services.AddOpenApi(); - -builder.Services.AddCors(options => -{ - options.AddDefaultPolicy(policy => - { - policy.AllowAnyOrigin() - .AllowAnyMethod() - .AllowAnyHeader(); - }); -}); - -builder.Services.ConfigureHttpJsonOptions(options => -{ - options.SerializerOptions.PropertyNamingPolicy = JsonNamingPolicy.CamelCase; - options.SerializerOptions.DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull; - options.SerializerOptions.Converters.Add(new JsonStringEnumConverter(JsonNamingPolicy.CamelCase)); -}); +builder.Services.AddCSharpDbRestApi(); var app = builder.Build(); @@ -44,39 +23,9 @@ _ = await dbClient.GetInfoAsync(); } -// ─── Middleware pipeline ──────────────────────────────────── - -app.UseCors(); -app.UseMiddleware(); - -if (app.Environment.IsDevelopment()) -{ - app.MapOpenApi(); - app.MapScalarApiReference(options => - { - options.WithTitle("CSharpDB API"); - options.WithDefaultHttpClient(ScalarTarget.CSharp, ScalarClient.HttpClient); - }); -} - -// ─── Endpoints ────────────────────────────────────────────── - -var api = app.MapGroup("/api"); +// ─── Middleware pipeline and endpoints ────────────────────── -api.MapTableEndpoints(); -api.MapRowEndpoints(); -api.MapIndexEndpoints(); -api.MapViewEndpoints(); -api.MapTriggerEndpoints(); -api.MapProcedureEndpoints(); -api.MapSavedQueryEndpoints(); -api.MapSqlEndpoints(); -api.MapPipelineEndpoints(); -api.MapTransactionEndpoints(); -api.MapCollectionEndpoints(); -api.MapSchemaEndpoints(); -api.MapInspectEndpoints(); -api.MapMaintenanceEndpoints(); +app.MapCSharpDbRestApi(); app.Run(); diff --git a/src/CSharpDB.Api/README.md b/src/CSharpDB.Api/README.md index aae7138e..244ecd12 100644 --- a/src/CSharpDB.Api/README.md +++ b/src/CSharpDB.Api/README.md @@ -6,18 +6,23 @@ It is a thin ASP.NET Core layer over `CSharpDB.Client`. Requests are handled through `ICSharpDbClient`, which currently uses the direct engine-backed client under the hood. -gRPC is not hosted here. The dedicated gRPC host lives in -`CSharpDB.Daemon`. +The standalone API host remains supported for REST-only deployments. For new +remote deployments that need both REST and gRPC, prefer +[`CSharpDB.Daemon`](../CSharpDB.Daemon/README.md), which now hosts the same +REST `/api` surface and gRPC from one warm database instance. ## What This Project Is For Use this project when you want to: -- expose a local CSharpDB database over HTTP +- expose a local CSharpDB database over HTTP without running the daemon - test the database through a browser-based API UI - integrate with tools that prefer REST over direct embedded access - inspect database, WAL, and index state remotely +Use `CSharpDB.Daemon` when REST and gRPC clients should share one long-running +remote database host. + Use `CSharpDB.Client` directly when you are writing an in-process consumer and do not need HTTP. @@ -30,6 +35,8 @@ The API host is intentionally thin: - `ICSharpDbClient` is registered at startup from configuration - the client is warmed up during startup with `GetInfoAsync()` so configuration and database initialization failures happen early +- the route/middleware setup is shared with `CSharpDB.Daemon` so both hosts + expose the same REST API surface Current request flow: @@ -446,8 +453,10 @@ Current status mapping: - the API is currently mapped under `/api`, not `/api/v1` - there is no dedicated endpoint yet for creating tables outside raw SQL - the API uses the same authoritative client contract as other consumers -- the API host is suitable for local development and integration testing, but it - is not yet production-hardened +- the standalone API host is suitable for local development, REST-only hosting, + and integration testing, but it is not yet production-hardened +- use `CSharpDB.Daemon` when REST should share a warm hosted database instance + with gRPC ## Useful Commands diff --git a/src/CSharpDB.Client/CSharpDB.Client.csproj b/src/CSharpDB.Client/CSharpDB.Client.csproj index a7677c20..c35ba89e 100644 --- a/src/CSharpDB.Client/CSharpDB.Client.csproj +++ b/src/CSharpDB.Client/CSharpDB.Client.csproj @@ -22,11 +22,12 @@ + all runtime; build; native; contentfiles; analyzers; buildtransitive - + diff --git a/src/CSharpDB.Client/Internal/GrpcTransportClient.cs b/src/CSharpDB.Client/Internal/GrpcTransportClient.cs index 47edba72..85ab777f 100644 --- a/src/CSharpDB.Client/Internal/GrpcTransportClient.cs +++ b/src/CSharpDB.Client/Internal/GrpcTransportClient.cs @@ -4,6 +4,7 @@ using CSharpDB.Storage.Diagnostics; using Grpc.Core; using Grpc.Net.Client; +using Grpc.Net.Client.Web; using CoreDbException = CSharpDB.Primitives.CSharpDbException; using CoreErrorCode = CSharpDB.Primitives.ErrorCode; using Empty = Google.Protobuf.WellKnownTypes.Empty; @@ -30,6 +31,14 @@ public GrpcTransportClient(Uri endpoint, HttpClient? httpClient = null) channelOptions.HttpClient = httpClient; channelOptions.DisposeHttpClient = false; } + else if (endpoint.Scheme.Equals(Uri.UriSchemeHttp, StringComparison.OrdinalIgnoreCase)) + { + channelOptions.HttpHandler = new GrpcWebHandler( + GrpcWebMode.GrpcWeb, + new HttpClientHandler()); + channelOptions.HttpVersion = System.Net.HttpVersion.Version11; + channelOptions.HttpVersionPolicy = HttpVersionPolicy.RequestVersionExact; + } _channel = GrpcChannel.ForAddress(endpoint, channelOptions); _client = new CSharpDbRpc.CSharpDbRpcClient(_channel); diff --git a/src/CSharpDB.Daemon/CSharpDB.Daemon.csproj b/src/CSharpDB.Daemon/CSharpDB.Daemon.csproj index 485da44a..df5bddb2 100644 --- a/src/CSharpDB.Daemon/CSharpDB.Daemon.csproj +++ b/src/CSharpDB.Daemon/CSharpDB.Daemon.csproj @@ -9,11 +9,21 @@ + + + + + + + + + + diff --git a/src/CSharpDB.Daemon/Program.cs b/src/CSharpDB.Daemon/Program.cs index ec249269..85a199d8 100644 --- a/src/CSharpDB.Daemon/Program.cs +++ b/src/CSharpDB.Daemon/Program.cs @@ -1,8 +1,16 @@ +using CSharpDB.Api; using CSharpDB.Client; using CSharpDB.Daemon.Configuration; using CSharpDB.Daemon.Grpc; var builder = WebApplication.CreateBuilder(args); +bool enableRestApi = builder.Configuration.GetValue("CSharpDB:Daemon:EnableRestApi", true); + +builder.Host.UseWindowsService(options => +{ + options.ServiceName = "CSharpDB Daemon"; +}); +builder.Host.UseSystemd(); builder.Services.AddSingleton(sp => DaemonClientOptionsBuilder.BindHostDatabaseOptions(sp.GetRequiredService())); @@ -14,6 +22,11 @@ builder.Services.AddCSharpDbClient(sp => sp.GetRequiredService()); +if (enableRestApi) +{ + builder.Services.AddCSharpDbRestApi(); +} + builder.Services.AddGrpc(); var app = builder.Build(); @@ -24,7 +37,17 @@ _ = await dbClient.GetInfoAsync(); } -app.MapGrpcService(); +if (app.Configuration.GetValue("CSharpDB:Daemon:EnableRestApi", true)) +{ + app.MapCSharpDbRestApi(options => + { + options.OpenApiTitle = "CSharpDB Daemon API"; + options.ApplyMiddlewareToApiOnly = true; + }); +} + +app.UseGrpcWeb(); +app.MapGrpcService().EnableGrpcWeb(); app.Run(); diff --git a/src/CSharpDB.Daemon/README.md b/src/CSharpDB.Daemon/README.md index 8619d153..1d44ea52 100644 --- a/src/CSharpDB.Daemon/README.md +++ b/src/CSharpDB.Daemon/README.md @@ -1,20 +1,22 @@ # CSharpDB.Daemon -`CSharpDB.Daemon` is the dedicated gRPC host for CSharpDB. +`CSharpDB.Daemon` is the preferred combined remote host for CSharpDB. -It exposes the `CSharpDB.Client` contract over gRPC and keeps that transport -separate from the REST-only [`CSharpDB.Api`](../CSharpDB.Api/README.md) host. +It exposes the `CSharpDB.Client` contract over gRPC and the existing REST/HTTP +surface under `/api` from one long-running process. Both transports use the same +warm daemon-hosted `ICSharpDbClient` instance. ## What This Process Is Designed For `CSharpDB.Daemon` is designed as a long-running service process for systems that want to keep one CSharpDB database available to multiple clients through a -stable gRPC endpoint. +stable remote endpoint. Current design assumptions: - one daemon process manages one database file -- clients talk to it through `CSharpDB.Client` with `Transport = Grpc` +- clients talk to it through `CSharpDB.Client` with `Transport = Grpc` or + `Transport = Http` - the daemon is best suited for trusted internal networks, local development, or service-to-service communication - the daemon is a thin transport host over `ICSharpDbClient`, not a separate @@ -24,16 +26,17 @@ This makes it a good fit for: - backend services that should not open the database file directly - local tools that need a reusable database process +- REST tooling and gRPC clients that should share one warm database instance - test environments that want a stable remote-style endpoint -- future service-daemon work without coupling that work to the REST API host +- future service-daemon work without adding a second database host process It is not yet designed as: - a multi-tenant database server - a public internet-facing database endpoint - a multi-database host in one process -- a hardened production service with built-in auth, metrics, health endpoints, - or OS service installers +- a hardened public production service with built-in auth, metrics, or health + endpoints ## Current Runtime Model @@ -43,7 +46,8 @@ Today the daemon does the following: 2. registers a direct `ICSharpDbClient` from configuration 3. opens and validates the configured database during startup by calling `GetInfoAsync()` -4. exposes explicit generated gRPC methods under +4. exposes REST routes under `/api`, including `/api/info` +5. exposes explicit generated gRPC methods under `/csharpdb.rpc.CSharpDbRpc/*` such as `/csharpdb.rpc.CSharpDbRpc/GetInfo` and `/csharpdb.rpc.CSharpDbRpc/ExecuteSql` @@ -67,17 +71,17 @@ The contract is now method-based and strongly typed: - dynamic row/document/argument values use a recursive protobuf value shape instead of JSON payload strings - maintenance operations such as backup/restore, reindex, vacuum, and foreign-key retrofit migration are first-class RPCs rather than a generic tunnel -## Protocol Boundary +## Protocol Model -The host split is intentional: +`CSharpDB.Daemon` now hosts both remote protocols by default: -- `CSharpDB.Api` is the REST/HTTP host -- `CSharpDB.Daemon` is the gRPC host +- REST/HTTP routes are available under `/api` +- gRPC routes are available under `/csharpdb.rpc.CSharpDbRpc/*` +- both transports share the same daemon-hosted database client and cache -If you want HTTP/JSON routes under `/api/...`, use `CSharpDB.Api`. - -If you want the `CSharpDB.Client` remote transport over gRPC, use -`CSharpDB.Daemon`. +The standalone [`CSharpDB.Api`](../CSharpDB.Api/README.md) host remains +supported for existing REST-only deployments. Use the daemon when REST and gRPC +clients should share one warm database instance. ## Transport Guidance @@ -85,11 +89,16 @@ Use transports as follows: - `Direct`: fastest overall when the caller can open the database locally in the same process space; this bypasses the daemon entirely - `Grpc`: recommended remote transport and the fastest supported network transport in the current codebase -- `Http`: use [`CSharpDB.Api`](../CSharpDB.Api/README.md) when you need REST/JSON routes rather than the client RPC contract +- `Http`: use the daemon for REST/JSON routes that should share the daemon database instance, or use [`CSharpDB.Api`](../CSharpDB.Api/README.md) for standalone REST-only hosting - `NamedPipes`: reserved in transport enums and parsers, but not implemented end to end today -For remote access to a daemon on another machine, use `Grpc` with a normal base -address such as: +When `Transport = Grpc` connects to a plain `http://` daemon endpoint, +`CSharpDB.Client` uses gRPC-Web compatibility so gRPC and REST can share the +default HTTP service URL. HTTPS endpoints and custom gRPC test clients can still +use native gRPC. + +For remote access to a daemon on another machine, use `Grpc` or `Http` with a +normal base address such as: ```text http://db-host:5820 @@ -109,10 +118,11 @@ For best remote performance: ## Requirements - .NET SDK 10.0 or newer to build from source +- no .NET SDK requirement when using the self-contained release archives - a filesystem location where the daemon can create and update: - `*.db` - `*.wal` -- an HTTP/2-capable path between client and daemon +- an HTTP/2-capable path between client and daemon when using native gRPC For local development, plain `http://localhost:...` is enough. @@ -125,6 +135,7 @@ The daemon reads the database path plus a small daemon-only host database section: - `ConnectionStrings:CSharpDB` +- `CSharpDB:Daemon:EnableRestApi` - `CSharpDB:HostDatabase:OpenMode` - `CSharpDB:HostDatabase:ImplicitInsertExecutionMode` - `CSharpDB:HostDatabase:UseWriteOptimizedPreset` @@ -142,6 +153,9 @@ Default [`appsettings.json`](./appsettings.json): "CSharpDB": "Data Source=csharpdb.db" }, "CSharpDB": { + "Daemon": { + "EnableRestApi": true + }, "HostDatabase": { "OpenMode": "HybridIncrementalDurable", "ImplicitInsertExecutionMode": "ConcurrentWriteTransactions", @@ -161,6 +175,7 @@ Data Source=csharpdb.db Current daemon defaults: +- `EnableRestApi = true` - `OpenMode = HybridIncrementalDurable` - `ImplicitInsertExecutionMode = ConcurrentWriteTransactions` - `UseWriteOptimizedPreset = true` @@ -171,6 +186,7 @@ Useful overrides: ```powershell $env:ConnectionStrings__CSharpDB = "Data Source=C:\\data\\app.db" +$env:CSharpDB__Daemon__EnableRestApi = "false" $env:CSharpDB__HostDatabase__OpenMode = "Direct" $env:CSharpDB__HostDatabase__ImplicitInsertExecutionMode = "Serialized" $env:CSharpDB__HostDatabase__HotTableNames__0 = "users" @@ -182,6 +198,7 @@ Linux/macOS shell: ```bash export ConnectionStrings__CSharpDB="Data Source=/var/lib/csharpdb/app.db" +export CSharpDB__Daemon__EnableRestApi="false" export CSharpDB__HostDatabase__OpenMode="Direct" export CSharpDB__HostDatabase__ImplicitInsertExecutionMode="Serialized" export CSharpDB__HostDatabase__HotTableNames__0="users" @@ -192,7 +209,9 @@ export ASPNETCORE_URLS="http://0.0.0.0:5820" `OpenMode=HybridIncrementalDurable` is the default and recommended daemon shape. Use `Direct` only when you want the host to open the backing file without the lazy-resident hybrid cache. `HotTableNames` and -`HotCollectionNames` are optional hybrid-only preload hints. +`HotCollectionNames` are optional hybrid-only preload hints. Set +`CSharpDB:Daemon:EnableRestApi=false` only when the daemon should expose gRPC +without the REST `/api` surface. ## Local Development @@ -220,7 +239,7 @@ scripts documented in [`scripts/README.md`](../../scripts/README.md). The intended consumer is `CSharpDB.Client`. -Example: +gRPC example: ```csharp using CSharpDB.Client; @@ -235,11 +254,26 @@ var info = await client.GetInfoAsync(); var tables = await client.GetTableNamesAsync(); ``` +HTTP/REST example against the same daemon: + +```csharp +using CSharpDB.Client; + +await using var client = CSharpDbClient.Create(new CSharpDbClientOptions +{ + Transport = CSharpDbTransport.Http, + Endpoint = "http://localhost:5820", +}); + +var info = await client.GetInfoAsync(); +``` + Notes: -- use `Transport = Grpc` +- use `Transport = Grpc` for the generated RPC contract +- use `Transport = Http` for REST/JSON routes under `/api` - set `Endpoint` to the daemon base address, not the raw RPC path -- the client handles the generated gRPC contract internally +- the client handles the generated gRPC and REST contracts internally - for remote hosts, prefer `https://...` or private-network `http://...` endpoints with long-lived client reuse ## Deployment Patterns @@ -292,6 +326,14 @@ Self-contained Linux publish: dotnet publish src/CSharpDB.Daemon/CSharpDB.Daemon.csproj -c Release -r linux-x64 --self-contained true -o ./artifacts/daemon-linux-x64 ``` +Release archive publish: + +```powershell +.\scripts\Publish-CSharpDbDaemonRelease.ps1 -Version 3.4.0 -Runtime win-x64 +.\scripts\Publish-CSharpDbDaemonRelease.ps1 -Version 3.4.0 -Runtime linux-x64 +.\scripts\Publish-CSharpDbDaemonRelease.ps1 -Version 3.4.0 -Runtime osx-arm64 +``` + ### 3. Sidecar-Style Deployment Best when: @@ -305,58 +347,155 @@ Typical pattern: - bind to localhost or a private container port - point only the colocated app at the daemon -## Example Service Setup +## Release Archives + +Tagged releases now attach self-contained daemon archives: -The repo does not yet ship production-ready install scripts for systemd, -Windows Service, or launchd. Current deployment is manual. +- `csharpdb-daemon-v{version}-win-x64.zip` +- `csharpdb-daemon-v{version}-linux-x64.tar.gz` +- `csharpdb-daemon-v{version}-osx-arm64.tar.gz` +- `SHA256SUMS.txt` -Use the following as starting points, not as final supported installers. +Each archive contains the daemon executable, production-ready default config, +this README, and service install assets under `service/`. + +Verify an archive before installing: + +```bash +sha256sum -c SHA256SUMS.txt +``` + +For a maintainer/operator walkthrough that explains which scripts are part of +the release cycle and which scripts are run after downloading an archive, see +[`scripts/README.md`](../../scripts/README.md). + +## Example Service Setup + +The release archives include service scripts and templates. The scripts copy the +extracted archive into an install directory, write `appsettings.Production.json`, +configure the database path and bind URL, and register the OS service. ### Windows Service -Publish first, then register the executable: +Run from an elevated PowerShell session inside the extracted Windows archive: + +```powershell +.\service\windows\install-csharpdb-daemon.ps1 -Start +``` + +Defaults: + +- service name: `CSharpDBDaemon` +- install directory: `C:\Program Files\CSharpDB\Daemon` +- data directory: `C:\ProgramData\CSharpDB` +- bind URL: `http://127.0.0.1:5820` + +Override defaults: ```powershell -sc.exe create CSharpDBDaemon binPath= "C:\Services\CSharpDB\CSharpDB.Daemon.exe" -sc.exe start CSharpDBDaemon +.\service\windows\install-csharpdb-daemon.ps1 ` + -ServiceName CSharpDBDaemon ` + -InstallDirectory "C:\Services\CSharpDB" ` + -DataDirectory "D:\Data\CSharpDB" ` + -Url "http://0.0.0.0:5820" ` + -Force ` + -Start ``` -Recommended: +Uninstall: -- set `ConnectionStrings__CSharpDB` as a machine or service environment value -- run under a dedicated account with access only to the database directory -- bind to a private address where possible +```powershell +.\service\windows\uninstall-csharpdb-daemon.ps1 +``` ### systemd -Example unit: +Run from inside the extracted Linux archive: + +```bash +sudo ./service/linux/install-csharpdb-daemon.sh --start +``` -```ini -[Unit] -Description=CSharpDB gRPC Daemon -After=network.target +Defaults: -[Service] -WorkingDirectory=/opt/csharpdb -ExecStart=/opt/csharpdb/CSharpDB.Daemon -Environment=ConnectionStrings__CSharpDB=Data Source=/var/lib/csharpdb/app.db -Environment=ASPNETCORE_URLS=http://0.0.0.0:5820 -Restart=on-failure -User=csharpdb -Group=csharpdb +- service name: `csharpdb-daemon` +- install directory: `/opt/csharpdb-daemon` +- data directory: `/var/lib/csharpdb` +- service user/group: `csharpdb` +- bind URL: `http://127.0.0.1:5820` -[Install] -WantedBy=multi-user.target +Override defaults: + +```bash +sudo ./service/linux/install-csharpdb-daemon.sh \ + --install-dir /srv/csharpdb-daemon \ + --data-dir /srv/csharpdb-data \ + --url http://0.0.0.0:5820 \ + --force \ + --start +``` + +Uninstall: + +```bash +sudo ./service/linux/uninstall-csharpdb-daemon.sh ``` ### launchd -The same model applies on macOS: +Run from inside the extracted macOS archive: + +```bash +sudo ./service/macos/install-csharpdb-daemon.sh --start +``` + +Defaults: + +- service label: `com.csharpdb.daemon` +- install directory: `/usr/local/lib/csharpdb-daemon` +- data directory: `/usr/local/var/csharpdb` +- bind URL: `http://127.0.0.1:5820` + +Override defaults: -- publish the daemon -- register it with `launchd` -- provide `ConnectionStrings__CSharpDB` and `ASPNETCORE_URLS` -- run it as a dedicated service user where appropriate +```bash +sudo ./service/macos/install-csharpdb-daemon.sh \ + --install-dir /usr/local/lib/csharpdb-daemon \ + --data-dir /usr/local/var/csharpdb \ + --url http://0.0.0.0:5820 \ + --force \ + --start +``` + +Uninstall: + +```bash +sudo ./service/macos/uninstall-csharpdb-daemon.sh +``` + +## Upgrade And Configuration + +To upgrade, extract the newer archive and rerun the matching install script with +the same service name, data directory, and `--force` / `-Force`. The scripts +replace the installed daemon files but do not delete the database directory. + +The generated production config keeps the current daemon defaults: + +- `OpenMode = HybridIncrementalDurable` +- `ImplicitInsertExecutionMode = ConcurrentWriteTransactions` +- `UseWriteOptimizedPreset = true` + +Service-level environment variables still override JSON configuration. The +supported keys remain the standard daemon settings: + +- `ConnectionStrings__CSharpDB` +- `ASPNETCORE_URLS` +- `CSharpDB__Daemon__EnableRestApi` +- `CSharpDB__HostDatabase__OpenMode` +- `CSharpDB__HostDatabase__ImplicitInsertExecutionMode` +- `CSharpDB__HostDatabase__UseWriteOptimizedPreset` +- `CSharpDB__HostDatabase__HotTableNames__0` +- `CSharpDB__HostDatabase__HotCollectionNames__0` ## Storage And Filesystem Notes @@ -373,14 +512,17 @@ Operational guidance: ## Networking Notes -gRPC requires HTTP/2 semantics. +Native gRPC requires HTTP/2 semantics. For the default plaintext HTTP service +URL, `CSharpDB.Client` uses gRPC-Web compatibility for `Transport = Grpc` so +gRPC and REST can share the same base URL. REST uses ordinary HTTP semantics +under `/api`. Practical guidance: - local development can use `http://localhost:...` - internal deployments should prefer private networking - if you place the daemon behind a proxy or ingress, make sure that proxy - correctly supports gRPC + correctly supports native gRPC or gRPC-Web, depending on the client path - do not expose the daemon directly to untrusted clients until authentication, authorization, and stronger operational controls exist @@ -390,6 +532,7 @@ Current state: - startup fails early if the database cannot be opened - ASP.NET Core logging is available through the standard host logging pipeline +- `/api/info` is available when REST hosting is enabled Not implemented yet in this host: @@ -398,11 +541,10 @@ Not implemented yet in this host: - authentication - authorization - TLS-specific configuration helpers -- admin endpoints -- service install scripts +- admin endpoints beyond the existing database REST API For broader future direction, see -[`docs/service-daemon/README.md`](../../docs/service-daemon/README.md). +[`docs/roadmap.md`](../../docs/roadmap.md). ## Troubleshooting @@ -412,9 +554,14 @@ Check: - the daemon is running - `Endpoint` points at the daemon base address -- `Transport = Grpc` is set in `CSharpDbClientOptions` +- `Transport = Grpc` or `Transport = Http` is set in `CSharpDbClientOptions` - the bound URL in `ASPNETCORE_URLS` is reachable from the client +### REST returns 404 under `/api` + +Check whether `CSharpDB:Daemon:EnableRestApi` or +`CSharpDB__Daemon__EnableRestApi` has been set to `false`. + ### Startup fails immediately Check: @@ -437,11 +584,13 @@ Important files: - [`Program.cs`](./Program.cs) - daemon host startup and service wiring - [`Grpc/CSharpDbRpcService.cs`](./Grpc/CSharpDbRpcService.cs) - gRPC server implementation +- [`../CSharpDB.Api/CSharpDbRestApiHostExtensions.cs`](../CSharpDB.Api/CSharpDbRestApiHostExtensions.cs) - shared REST host wiring - [`appsettings.json`](./appsettings.json) - default configuration - [`../CSharpDB.Client/Protos/csharpdb_rpc.proto`](../CSharpDB.Client/Protos/csharpdb_rpc.proto) - transport contract ## Status -This README documents the current daemon implementation, not the full planned -service-daemon feature set. The broader roadmap remains in -[`docs/service-daemon/README.md`](../../docs/service-daemon/README.md). +This README documents the current daemon implementation, v3.4.0 service +packaging, and v3.4.0 REST/gRPC host consolidation. Auth, TLS helpers, and +marketplace distribution remain tracked in +[`docs/roadmap.md`](../../docs/roadmap.md). diff --git a/src/CSharpDB.Daemon/appsettings.json b/src/CSharpDB.Daemon/appsettings.json index 7059a490..986e24ae 100644 --- a/src/CSharpDB.Daemon/appsettings.json +++ b/src/CSharpDB.Daemon/appsettings.json @@ -9,6 +9,9 @@ "CSharpDB": "Data Source=csharpdb.db" }, "CSharpDB": { + "Daemon": { + "EnableRestApi": true + }, "HostDatabase": { "OpenMode": "HybridIncrementalDurable", "ImplicitInsertExecutionMode": "ConcurrentWriteTransactions", diff --git a/src/CSharpDB.EntityFrameworkCore/CSharpDB.EntityFrameworkCore.csproj b/src/CSharpDB.EntityFrameworkCore/CSharpDB.EntityFrameworkCore.csproj index 39aab2d0..511e55f5 100644 --- a/src/CSharpDB.EntityFrameworkCore/CSharpDB.EntityFrameworkCore.csproj +++ b/src/CSharpDB.EntityFrameworkCore/CSharpDB.EntityFrameworkCore.csproj @@ -14,11 +14,11 @@ - + all runtime; build; native; contentfiles; analyzers; buildtransitive - + diff --git a/src/CSharpDB.Generators/CSharpDB.Generators.csproj b/src/CSharpDB.Generators/CSharpDB.Generators.csproj index f6fa6e10..800caaa7 100644 --- a/src/CSharpDB.Generators/CSharpDB.Generators.csproj +++ b/src/CSharpDB.Generators/CSharpDB.Generators.csproj @@ -16,7 +16,7 @@ - + diff --git a/src/CSharpDB.Mcp/CSharpDB.Mcp.csproj b/src/CSharpDB.Mcp/CSharpDB.Mcp.csproj index da230a28..2f9ef266 100644 --- a/src/CSharpDB.Mcp/CSharpDB.Mcp.csproj +++ b/src/CSharpDB.Mcp/CSharpDB.Mcp.csproj @@ -15,7 +15,7 @@ - + diff --git a/tests/CSharpDB.Admin.Forms.Tests/Admin/AdminClientOptionsBuilderTests.cs b/tests/CSharpDB.Admin.Forms.Tests/Admin/AdminClientOptionsBuilderTests.cs new file mode 100644 index 00000000..52e90d92 --- /dev/null +++ b/tests/CSharpDB.Admin.Forms.Tests/Admin/AdminClientOptionsBuilderTests.cs @@ -0,0 +1,120 @@ +using CSharpDB.Admin.Configuration; +using CSharpDB.Client; +using CSharpDB.Engine; +using Microsoft.Extensions.Configuration; + +namespace CSharpDB.Admin.Forms.Tests.Admin; + +public sealed class AdminClientOptionsBuilderTests +{ + [Fact] + public void Build_LocalDirectDefaultsToHybridIncrementalDurable() + { + IConfiguration configuration = CreateConfiguration(new Dictionary + { + ["ConnectionStrings:CSharpDB"] = "Data Source=admin.db", + }); + + AdminHostDatabaseOptions hostOptions = AdminClientOptionsBuilder.BindHostDatabaseOptions(configuration); + + CSharpDbClientOptions options = AdminClientOptionsBuilder.Build( + configuration, + hostOptions, + CSharpDbTransport.Direct, + endpoint: null); + + Assert.Equal(CSharpDbTransport.Direct, options.Transport); + Assert.Equal("Data Source=admin.db", options.ConnectionString); + Assert.Null(options.Endpoint); + Assert.NotNull(options.DirectDatabaseOptions); + Assert.NotNull(options.HybridDatabaseOptions); + Assert.Equal( + ImplicitInsertExecutionMode.ConcurrentWriteTransactions, + options.DirectDatabaseOptions.ImplicitInsertExecutionMode); + Assert.Equal(HybridPersistenceMode.IncrementalDurable, options.HybridDatabaseOptions.PersistenceMode); + } + + [Fact] + public void Build_DirectOpenModeDisablesHybridOptions() + { + IConfiguration configuration = CreateConfiguration(new Dictionary + { + ["ConnectionStrings:CSharpDB"] = "Data Source=admin.db", + ["CSharpDB:HostDatabase:OpenMode"] = "Direct", + }); + + AdminHostDatabaseOptions hostOptions = AdminClientOptionsBuilder.BindHostDatabaseOptions(configuration); + + CSharpDbClientOptions options = AdminClientOptionsBuilder.Build( + configuration, + hostOptions, + CSharpDbTransport.Direct, + endpoint: null); + + Assert.Equal(AdminHostOpenMode.Direct, hostOptions.OpenMode); + Assert.NotNull(options.DirectDatabaseOptions); + Assert.Null(options.HybridDatabaseOptions); + } + + [Fact] + public void Build_RemoteEndpointDoesNotAttachDirectOrHybridOptions() + { + IConfiguration configuration = CreateConfiguration(new Dictionary + { + ["ConnectionStrings:CSharpDB"] = "Data Source=admin.db", + }); + AdminHostDatabaseOptions hostOptions = AdminClientOptionsBuilder.BindHostDatabaseOptions(configuration); + + CSharpDbClientOptions options = AdminClientOptionsBuilder.Build( + configuration, + hostOptions, + CSharpDbTransport.Grpc, + "http://127.0.0.1:5820"); + + Assert.Equal(CSharpDbTransport.Grpc, options.Transport); + Assert.Equal("http://127.0.0.1:5820", options.Endpoint); + Assert.Null(options.ConnectionString); + Assert.Null(options.DirectDatabaseOptions); + Assert.Null(options.HybridDatabaseOptions); + } + + [Fact] + public void Build_DirectEndpointUsesHybridOptions() + { + IConfiguration configuration = CreateConfiguration(new Dictionary()); + AdminHostDatabaseOptions hostOptions = AdminClientOptionsBuilder.BindHostDatabaseOptions(configuration); + + CSharpDbClientOptions options = AdminClientOptionsBuilder.Build( + configuration, + hostOptions, + transport: null, + endpoint: "endpoint.db"); + + Assert.Null(options.Transport); + Assert.Equal("endpoint.db", options.Endpoint); + Assert.Null(options.ConnectionString); + Assert.NotNull(options.DirectDatabaseOptions); + Assert.NotNull(options.HybridDatabaseOptions); + } + + [Fact] + public void BuildDirectDataSource_UsesHybridOptionsForDatabaseSwitches() + { + AdminHostDatabaseOptions hostOptions = new(); + + CSharpDbClientOptions options = AdminClientOptionsBuilder.BuildDirectDataSource( + @"C:\data\switched.db", + hostOptions); + + Assert.Equal(CSharpDbTransport.Direct, options.Transport); + Assert.Equal(@"C:\data\switched.db", options.DataSource); + Assert.NotNull(options.DirectDatabaseOptions); + Assert.NotNull(options.HybridDatabaseOptions); + Assert.Equal(HybridPersistenceMode.IncrementalDurable, options.HybridDatabaseOptions.PersistenceMode); + } + + private static IConfiguration CreateConfiguration(Dictionary values) + => new ConfigurationBuilder() + .AddInMemoryCollection(values) + .Build(); +} diff --git a/tests/CSharpDB.Admin.Forms.Tests/Components/Shared/DataGridTests.cs b/tests/CSharpDB.Admin.Forms.Tests/Components/Shared/DataGridTests.cs index a1625240..88428733 100644 --- a/tests/CSharpDB.Admin.Forms.Tests/Components/Shared/DataGridTests.cs +++ b/tests/CSharpDB.Admin.Forms.Tests/Components/Shared/DataGridTests.cs @@ -40,6 +40,150 @@ public async Task ApplyFilterDraftAsync_EmptyDraftClearsActiveFilter() Assert.Single(GetRows(component)); } + [Fact] + public void BuildWhereClause_DefaultFilterModeUsesContainsLikePredicate() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Name" }); + GetFilters(component)[0] = "Ali"; + + string whereClause = (string)InvokeNonPublic(component, "BuildWhereClause")!; + + Assert.Equal(" WHERE TEXT(Name) LIKE '%Ali%' ESCAPE '!'", whereClause); + } + + [Fact] + public void BuildWhereClause_StartsWithFilterModeUsesRightWildcardOnly() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Name" }); + GetFilters(component)[0] = "Ali"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.StartsWith; + + string whereClause = (string)InvokeNonPublic(component, "BuildWhereClause")!; + + Assert.Equal(" WHERE TEXT(Name) LIKE 'Ali%' ESCAPE '!'", whereClause); + } + + [Fact] + public void BuildWhereClause_EndsWithFilterModeUsesLeftWildcardOnly() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Name" }); + GetFilters(component)[0] = "ice"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.EndsWith; + + string whereClause = (string)InvokeNonPublic(component, "BuildWhereClause")!; + + Assert.Equal(" WHERE TEXT(Name) LIKE '%ice' ESCAPE '!'", whereClause); + } + + [Fact] + public void BuildWhereClause_ExactTextFilterModeUsesSargableColumnPredicate() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Name" }); + SetProperty(component, nameof(DataGrid.ColumnTypes), new[] { "TEXT" }); + GetFilters(component)[0] = "Alice"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.Exact; + + string whereClause = (string)InvokeNonPublic(component, "BuildWhereClause")!; + + Assert.Equal(" WHERE Name = 'Alice'", whereClause); + } + + [Fact] + public void BuildWhereClause_ExactIntegerFilterModeUsesSargableNumericPredicate() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Id" }); + SetProperty(component, nameof(DataGrid.ColumnTypes), new[] { "INTEGER" }); + GetFilters(component)[0] = "42"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.Exact; + + string whereClause = (string)InvokeNonPublic(component, "BuildWhereClause")!; + + Assert.Equal(" WHERE Id = 42", whereClause); + } + + [Fact] + public void BuildWhereClause_ExactIntegerFilterModeRejectsNonCanonicalDisplayValue() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Id" }); + SetProperty(component, nameof(DataGrid.ColumnTypes), new[] { "INTEGER" }); + GetFilters(component)[0] = "042"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.Exact; + + string whereClause = (string)InvokeNonPublic(component, "BuildWhereClause")!; + + Assert.Equal(" WHERE 0 = 1", whereClause); + } + + [Fact] + public void BuildWhereClause_ExactFilterModeWithUnknownTypeFallsBackToTextPredicate() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Name" }); + GetFilters(component)[0] = "Alice"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.Exact; + + string whereClause = (string)InvokeNonPublic(component, "BuildWhereClause")!; + + Assert.Equal(" WHERE TEXT(Name) = 'Alice'", whereClause); + } + + [Fact] + public void ApplyFiltersAndSort_ExactFilterModeRequiresFullValue() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Name" }); + SetField(component, "_loadedAllRows", true); + GetAllRows(component).AddRange( + [ + new DataGridRow(["Alice"]), + new DataGridRow(["Alicia"]), + new DataGridRow(["alice"]), + ]); + GetFilters(component)[0] = "Alice"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.Exact; + + InvokeNonPublic(component, "ApplyFiltersAndSort"); + + var rows = GetRows(component); + Assert.Single(rows); + Assert.Equal("Alice", rows[0].CurrentValues[0]); + } + + [Fact] + public void ApplyFiltersAndSort_LikePlacementModesMatchExpectedSide() + { + var component = new DataGrid(); + SetProperty(component, nameof(DataGrid.Columns), new[] { "Name" }); + SetField(component, "_loadedAllRows", true); + GetAllRows(component).AddRange( + [ + new DataGridRow(["Alice"]), + new DataGridRow(["Malice"]), + new DataGridRow(["Alicia"]), + ]); + GetFilters(component)[0] = "Ali"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.StartsWith; + + InvokeNonPublic(component, "ApplyFiltersAndSort"); + + var rows = GetRows(component); + Assert.Equal(["Alice", "Alicia"], rows.Select(row => (string)row.CurrentValues[0]!).ToArray()); + + GetFilters(component)[0] = "ice"; + GetFilterModes(component)[0] = DataGridFilterMatchMode.EndsWith; + + InvokeNonPublic(component, "ApplyFiltersAndSort"); + + rows = GetRows(component); + Assert.Equal(["Alice", "Malice"], rows.Select(row => (string)row.CurrentValues[0]!).ToArray()); + } + [Fact] public void QueryPagingWithoutExactTotal_ShowsRangeAndKeepsNextEnabled() { @@ -103,6 +247,9 @@ private static Dictionary GetFilters(DataGrid component) private static Dictionary GetFilterInputs(DataGrid component) => GetField>(component, "_filterInputs"); + private static Dictionary GetFilterModes(DataGrid component) + => GetField>(component, "_filterModes"); + private static List GetAllRows(DataGrid component) => GetField>(component, "_allRows"); diff --git a/tests/CSharpDB.Api.Tests/CSharpDB.Api.Tests.csproj b/tests/CSharpDB.Api.Tests/CSharpDB.Api.Tests.csproj index 838ad3f6..03e0076c 100644 --- a/tests/CSharpDB.Api.Tests/CSharpDB.Api.Tests.csproj +++ b/tests/CSharpDB.Api.Tests/CSharpDB.Api.Tests.csproj @@ -11,7 +11,7 @@ - + diff --git a/tests/CSharpDB.Benchmarks/CSharpDB.Benchmarks.csproj b/tests/CSharpDB.Benchmarks/CSharpDB.Benchmarks.csproj index 92425360..1efe096a 100644 --- a/tests/CSharpDB.Benchmarks/CSharpDB.Benchmarks.csproj +++ b/tests/CSharpDB.Benchmarks/CSharpDB.Benchmarks.csproj @@ -10,9 +10,9 @@ - - - + + + diff --git a/tests/CSharpDB.Daemon.Tests/CSharpDB.Daemon.Tests.csproj b/tests/CSharpDB.Daemon.Tests/CSharpDB.Daemon.Tests.csproj index 5b4662c9..d294d8a9 100644 --- a/tests/CSharpDB.Daemon.Tests/CSharpDB.Daemon.Tests.csproj +++ b/tests/CSharpDB.Daemon.Tests/CSharpDB.Daemon.Tests.csproj @@ -11,7 +11,7 @@ - + diff --git a/tests/CSharpDB.Daemon.Tests/DaemonPackagingAssetsTests.cs b/tests/CSharpDB.Daemon.Tests/DaemonPackagingAssetsTests.cs new file mode 100644 index 00000000..c2cb6ee1 --- /dev/null +++ b/tests/CSharpDB.Daemon.Tests/DaemonPackagingAssetsTests.cs @@ -0,0 +1,106 @@ +namespace CSharpDB.Daemon.Tests; + +public sealed class DaemonPackagingAssetsTests +{ + [Fact] + public void PublishScript_UsesExpectedDaemonArchiveContract() + { + string repoRoot = FindRepoRoot(); + string script = File.ReadAllText(Path.Combine(repoRoot, "scripts", "Publish-CSharpDbDaemonRelease.ps1")); + + Assert.Contains("win-x64", script); + Assert.Contains("linux-x64", script); + Assert.Contains("osx-arm64", script); + Assert.Contains("-p:PublishSingleFile=true", script); + Assert.Contains("-p:PublishTrimmed=false", script); + Assert.Contains("csharpdb-daemon-v$ReleaseVersion-$Rid", script); + Assert.Contains("SHA256SUMS.txt", script); + } + + [Fact] + public void ServiceInstallAssets_ContainExpectedDefaults() + { + string repoRoot = FindRepoRoot(); + + string windowsInstall = File.ReadAllText(Path.Combine( + repoRoot, + "deploy", + "daemon", + "windows", + "install-csharpdb-daemon.ps1")); + string linuxInstall = File.ReadAllText(Path.Combine( + repoRoot, + "deploy", + "daemon", + "linux", + "install-csharpdb-daemon.sh")); + string macInstall = File.ReadAllText(Path.Combine( + repoRoot, + "deploy", + "daemon", + "macos", + "install-csharpdb-daemon.sh")); + + Assert.Contains("CSharpDBDaemon", windowsInstall); + Assert.Contains("CSharpDB\\Daemon", windowsInstall); + Assert.Contains("CSharpDB", windowsInstall); + Assert.Contains("http://127.0.0.1:5820", windowsInstall); + Assert.Contains("CSharpDB__Daemon__EnableRestApi=true", windowsInstall); + + Assert.Contains("/opt/csharpdb-daemon", linuxInstall); + Assert.Contains("/var/lib/csharpdb", linuxInstall); + Assert.Contains("SERVICE_USER=\"csharpdb\"", linuxInstall); + Assert.Contains("http://127.0.0.1:5820", linuxInstall); + Assert.Contains("CSharpDB__Daemon__EnableRestApi=true", linuxInstall); + + Assert.Contains("com.csharpdb.daemon", macInstall); + Assert.Contains("/usr/local/lib/csharpdb-daemon", macInstall); + Assert.Contains("/usr/local/var/csharpdb", macInstall); + Assert.Contains("http://127.0.0.1:5820", macInstall); + Assert.Contains("\"EnableRestApi\": true", macInstall); + } + + [Fact] + public void ServiceTemplates_AreParameterizedForInstallScripts() + { + string repoRoot = FindRepoRoot(); + string systemdTemplate = File.ReadAllText(Path.Combine( + repoRoot, + "deploy", + "daemon", + "linux", + "csharpdb-daemon.service")); + string launchdTemplate = File.ReadAllText(Path.Combine( + repoRoot, + "deploy", + "daemon", + "macos", + "com.csharpdb.daemon.plist")); + + Assert.Contains("{{INSTALL_DIR}}", systemdTemplate); + Assert.Contains("{{ENV_FILE}}", systemdTemplate); + Assert.Contains("{{SERVICE_USER}}", systemdTemplate); + Assert.Contains("{{SERVICE_GROUP}}", systemdTemplate); + + Assert.Contains("{{SERVICE_NAME}}", launchdTemplate); + Assert.Contains("{{INSTALL_DIR}}", launchdTemplate); + Assert.Contains("{{DATABASE_PATH}}", launchdTemplate); + Assert.Contains("{{URL}}", launchdTemplate); + Assert.Contains("CSharpDB__Daemon__EnableRestApi", launchdTemplate); + } + + private static string FindRepoRoot() + { + DirectoryInfo? directory = new(AppContext.BaseDirectory); + + while (directory is not null) + { + if (File.Exists(Path.Combine(directory.FullName, "CSharpDB.slnx"))) + return directory.FullName; + + directory = directory.Parent; + } + + throw new DirectoryNotFoundException("Could not locate repository root from test base directory."); + } +} diff --git a/tests/CSharpDB.Daemon.Tests/GrpcClientTests.cs b/tests/CSharpDB.Daemon.Tests/GrpcClientTests.cs index d18b505c..b482a475 100644 --- a/tests/CSharpDB.Daemon.Tests/GrpcClientTests.cs +++ b/tests/CSharpDB.Daemon.Tests/GrpcClientTests.cs @@ -170,6 +170,82 @@ public async Task GrpcClient_RowCrud_RoundTripsPrimitiveValues() Assert.Equal(12.5, Assert.IsType(browse.Rows[0][2])); } + [Fact] + public async Task Daemon_RestApi_IsServedByDefault() + { + using var transportClient = CreateHttpTransportClient(); + await using var client = CreateHttpClient(transportClient); + + DatabaseInfo info = await client.GetInfoAsync(Ct); + + Assert.Equal(Path.GetFullPath(_dbPath), info.DataSource); + } + + [Fact] + public async Task Daemon_RestAndGrpcClients_ShareHostedDatabaseState() + { + using var grpcTransportClient = CreateGrpcHttpClient(); + await using var grpcClient = CreateGrpcClient(grpcTransportClient); + + using var httpTransportClient = CreateHttpTransportClient(); + await using var httpClient = CreateHttpClient(httpTransportClient); + + SqlExecutionResult createResult = await grpcClient.ExecuteSqlAsync( + "CREATE TABLE daemon_shared_users (id INTEGER PRIMARY KEY, name TEXT)", + Ct); + Assert.Null(createResult.Error); + + await grpcClient.InsertRowAsync( + "daemon_shared_users", + new Dictionary { ["id"] = 1L, ["name"] = "grpc" }, + Ct); + + Dictionary? grpcRowFromRest = await httpClient.GetRowByPkAsync( + "daemon_shared_users", + "id", + 1L, + Ct); + Assert.NotNull(grpcRowFromRest); + Assert.Equal("grpc", Assert.IsType(grpcRowFromRest!["name"])); + + await httpClient.InsertRowAsync( + "daemon_shared_users", + new Dictionary { ["id"] = 2L, ["name"] = "rest" }, + Ct); + + Dictionary? restRowFromGrpc = await grpcClient.GetRowByPkAsync( + "daemon_shared_users", + "id", + 2L, + Ct); + Assert.NotNull(restRowFromGrpc); + Assert.Equal("rest", Assert.IsType(restRowFromGrpc!["name"])); + + Assert.Equal(2, await grpcClient.GetRowCountAsync("daemon_shared_users", Ct)); + } + + [Fact] + public async Task Daemon_RestApiCanBeDisabledWithoutDisablingGrpc() + { + using var factory = new TestDaemonFactory( + _dbPath, + new Dictionary + { + ["CSharpDB:Daemon:EnableRestApi"] = "false", + }); + + using var httpClient = factory.CreateClient(); + using HttpResponseMessage restResponse = await httpClient.GetAsync("/api/info", Ct); + + Assert.Equal(HttpStatusCode.NotFound, restResponse.StatusCode); + + using var grpcTransportClient = CreateGrpcHttpClient(factory); + await using var grpcClient = CreateGrpcClient(grpcTransportClient); + + DatabaseInfo info = await grpcClient.GetInfoAsync(Ct); + Assert.Equal(Path.GetFullPath(_dbPath), info.DataSource); + } + [Fact] public async Task GrpcClient_Collections_RoundTripNestedDocuments() { @@ -519,6 +595,12 @@ private ICSharpDbClient CreateGrpcClient(HttpClient transportClient) private HttpClient CreateGrpcHttpClient() => CreateGrpcHttpClient(_factory); + private HttpClient CreateHttpTransportClient() + => _factory.CreateClient(new WebApplicationFactoryClientOptions + { + BaseAddress = new Uri("http://localhost"), + }); + private static HttpClient CreateGrpcHttpClient(TestDaemonFactory factory) { return new HttpClient(factory.Server.CreateHandler()) @@ -529,6 +611,14 @@ private static HttpClient CreateGrpcHttpClient(TestDaemonFactory factory) }; } + private static ICSharpDbClient CreateHttpClient(HttpClient transportClient) + => CSharpDbClient.Create(new CSharpDbClientOptions + { + Transport = CSharpDbTransport.Http, + Endpoint = "http://localhost", + HttpClient = transportClient, + }); + private static DaemonHostDatabaseOptions GetResolvedHostDatabaseOptions(TestDaemonFactory factory) => factory.Services.GetRequiredService(); diff --git a/tests/CSharpDB.Data.Tests/CSharpDB.Data.Tests.csproj b/tests/CSharpDB.Data.Tests/CSharpDB.Data.Tests.csproj index f11c7408..5d2c8ea0 100644 --- a/tests/CSharpDB.Data.Tests/CSharpDB.Data.Tests.csproj +++ b/tests/CSharpDB.Data.Tests/CSharpDB.Data.Tests.csproj @@ -16,8 +16,8 @@ - - + + diff --git a/tests/CSharpDB.EntityFrameworkCore.Tests/CSharpDB.EntityFrameworkCore.Tests.csproj b/tests/CSharpDB.EntityFrameworkCore.Tests/CSharpDB.EntityFrameworkCore.Tests.csproj index 851a5f4d..737e8b3b 100644 --- a/tests/CSharpDB.EntityFrameworkCore.Tests/CSharpDB.EntityFrameworkCore.Tests.csproj +++ b/tests/CSharpDB.EntityFrameworkCore.Tests/CSharpDB.EntityFrameworkCore.Tests.csproj @@ -15,11 +15,11 @@ - + all runtime; build; native; contentfiles; analyzers; buildtransitive - + From 7b5affc65619c04c2cdef6af6dfeaecbf2921198 Mon Sep 17 00:00:00 2001 From: Maximum Code Date: Sat, 25 Apr 2026 07:31:16 -0700 Subject: [PATCH 03/10] Fix forms web static assets and theme styling --- .../Components/App.razor | 8 +- .../Components/Pages/Home.razor | 8 ++ src/CSharpDB.Admin.Forms.Web/Program.cs | 2 + .../wwwroot/css/app.css | 121 ++++++++++++++---- .../wwwroot/js/forms-host.js | 37 ++++++ 5 files changed, 147 insertions(+), 29 deletions(-) diff --git a/src/CSharpDB.Admin.Forms.Web/Components/App.razor b/src/CSharpDB.Admin.Forms.Web/Components/App.razor index e220983a..b0abc163 100644 --- a/src/CSharpDB.Admin.Forms.Web/Components/App.razor +++ b/src/CSharpDB.Admin.Forms.Web/Components/App.razor @@ -1,10 +1,16 @@ - + CSharpDB Forms + diff --git a/src/CSharpDB.Admin.Forms.Web/Components/Pages/Home.razor b/src/CSharpDB.Admin.Forms.Web/Components/Pages/Home.razor index 29b0521b..12c8cf92 100644 --- a/src/CSharpDB.Admin.Forms.Web/Components/Pages/Home.razor +++ b/src/CSharpDB.Admin.Forms.Web/Components/Pages/Home.razor @@ -10,6 +10,14 @@

Forms Runtime

Runnable form host for @DbClient.DataSource

+
diff --git a/src/CSharpDB.Admin.Forms.Web/Program.cs b/src/CSharpDB.Admin.Forms.Web/Program.cs index 29a20ce5..5e575400 100644 --- a/src/CSharpDB.Admin.Forms.Web/Program.cs +++ b/src/CSharpDB.Admin.Forms.Web/Program.cs @@ -5,6 +5,8 @@ var builder = WebApplication.CreateBuilder(args); +builder.WebHost.UseStaticWebAssets(); + builder.Services.AddRazorComponents() .AddInteractiveServerComponents(); diff --git a/src/CSharpDB.Admin.Forms.Web/wwwroot/css/app.css b/src/CSharpDB.Admin.Forms.Web/wwwroot/css/app.css index 6744ac48..08edb090 100644 --- a/src/CSharpDB.Admin.Forms.Web/wwwroot/css/app.css +++ b/src/CSharpDB.Admin.Forms.Web/wwwroot/css/app.css @@ -1,15 +1,52 @@ +html[data-theme="dark"] { + color-scheme: dark; + --bg-primary: #1a1b26; + --bg-secondary: #16171f; + --bg-tertiary: #1f2029; + --bg-elevated: #24253a; + --bg-hover: #292a3e; + --bg-active: #33345a; + --border-color: #2a2b3d; + --border-light: #363750; + --text-primary: #c0caf5; + --text-secondary: #7982a9; + --text-muted: #565f89; + --accent-blue: #7aa2f7; + --accent-red: #f7768e; + --shadow-elevated: 0 4px 24px rgba(0,0,0,0.4); +} + +html[data-theme="light"] { + color-scheme: light; + --bg-primary: #ffffff; + --bg-secondary: #f4f5f7; + --bg-tertiary: #ebedf0; + --bg-elevated: #ffffff; + --bg-hover: #e8eaed; + --bg-active: #d3d8e0; + --border-color: #d1d5db; + --border-light: #e5e7eb; + --text-primary: #1e293b; + --text-secondary: #475569; + --text-muted: #94a3b8; + --accent-blue: #3b82f6; + --accent-red: #ef4444; + --shadow-elevated: 0 4px 24px rgba(0,0,0,0.08); +} + html, body { margin: 0; padding: 0; min-height: 100%; - background: #f3f5f7; - color: #16202a; + background: var(--bg-primary); + color: var(--text-primary); font-family: Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; } body { min-height: 100vh; + transition: background 0.2s ease, color 0.2s ease; } a { @@ -19,12 +56,14 @@ a { code { font-family: "JetBrains Mono", Consolas, monospace; font-size: 0.92em; + color: var(--text-primary); } .forms-home { min-height: 100vh; box-sizing: border-box; padding: 28px 32px 40px; + background: var(--bg-primary); } .forms-home__header { @@ -39,73 +78,99 @@ code { margin: 0; font-size: 28px; font-weight: 600; - color: #101820; + color: var(--text-primary); } .forms-home__header p { margin: 6px 0 0; - color: #51606f; + color: var(--text-secondary); font-size: 14px; } .forms-home__intro { max-width: 880px; margin-bottom: 20px; - color: #334454; + color: var(--text-secondary); font-size: 14px; line-height: 1.5; } .forms-home__summary { margin-bottom: 10px; - color: #51606f; + color: var(--text-muted); font-size: 13px; } .forms-home__state { padding: 16px 18px; - border: 1px solid #d4dbe3; - background: #fff; - color: #334454; + border: 1px solid var(--border-color); + background: var(--bg-elevated); + color: var(--text-secondary); font-size: 14px; + box-shadow: var(--shadow-elevated); } .forms-home__state--error { - border-color: #e2b3b3; - color: #8b2f2f; - background: #fff7f7; + border-color: color-mix(in srgb, var(--accent-red) 42%, var(--border-color)); + color: var(--accent-red); + background: color-mix(in srgb, var(--accent-red) 12%, var(--bg-elevated)); } -.forms-home__action { +.forms-home__action, +.forms-home__theme-toggle { display: inline-flex; align-items: center; justify-content: center; min-height: 32px; padding: 0 12px; - border: 1px solid #b9c6d3; + border: 1px solid var(--border-color); border-radius: 6px; - background: #fff; + background: var(--bg-elevated); text-decoration: none; font-size: 13px; font-weight: 500; - color: #16202a; + color: var(--text-primary); + cursor: pointer; + transition: background 0.12s ease, border-color 0.12s ease, color 0.12s ease; } -.forms-home__action:hover { - background: #eef3f8; +.forms-home__action:hover, +.forms-home__theme-toggle:hover { + background: var(--bg-hover); + border-color: var(--accent-blue); +} + +.forms-home__theme-toggle { + gap: 8px; + flex-shrink: 0; + font-family: inherit; +} + +.forms-home__theme-icon { + display: inline-flex; + align-items: center; + justify-content: center; + width: 18px; + height: 18px; + border-radius: 50%; + background: color-mix(in srgb, var(--accent-blue) 18%, transparent); + color: var(--accent-blue); + font-size: 11px; + font-weight: 700; } .forms-table { width: 100%; border-collapse: collapse; - background: #fff; - border: 1px solid #d4dbe3; + background: var(--bg-elevated); + border: 1px solid var(--border-color); + box-shadow: var(--shadow-elevated); } .forms-table th, .forms-table td { padding: 12px 14px; - border-bottom: 1px solid #e6ebf1; + border-bottom: 1px solid var(--border-light); text-align: left; vertical-align: middle; font-size: 14px; @@ -115,13 +180,13 @@ code { font-size: 12px; font-weight: 600; text-transform: uppercase; - letter-spacing: 0.04em; - color: #5a6876; - background: #f7f9fb; + letter-spacing: 0; + color: var(--text-secondary); + background: var(--bg-tertiary); } .forms-table tbody tr:hover { - background: #fafcff; + background: var(--bg-hover); } .forms-table__actions { @@ -153,7 +218,7 @@ code { } .forms-table tr { - border-bottom: 1px solid #e6ebf1; + border-bottom: 1px solid var(--border-light); } .forms-table td { @@ -168,7 +233,7 @@ code { font-size: 11px; font-weight: 600; text-transform: uppercase; - letter-spacing: 0.04em; - color: #5a6876; + letter-spacing: 0; + color: var(--text-muted); } } diff --git a/src/CSharpDB.Admin.Forms.Web/wwwroot/js/forms-host.js b/src/CSharpDB.Admin.Forms.Web/wwwroot/js/forms-host.js index bdd77f8e..04526029 100644 --- a/src/CSharpDB.Admin.Forms.Web/wwwroot/js/forms-host.js +++ b/src/CSharpDB.Admin.Forms.Web/wwwroot/js/forms-host.js @@ -1,3 +1,40 @@ +window.formsTheme = { + get: () => localStorage.getItem('csharpdb-theme') || 'dark', + + set: (theme) => { + const nextTheme = theme === 'light' ? 'light' : 'dark'; + localStorage.setItem('csharpdb-theme', nextTheme); + document.documentElement.setAttribute('data-theme', nextTheme); + window.formsTheme.updateControls(); + }, + + toggle: () => { + window.formsTheme.set(window.formsTheme.get() === 'dark' ? 'light' : 'dark'); + }, + + updateControls: () => { + const theme = window.formsTheme.get(); + const label = theme === 'dark' ? 'Dark' : 'Light'; + const icon = theme === 'dark' ? 'D' : 'L'; + + document.querySelectorAll('[data-theme-label]').forEach(element => { + element.textContent = label; + }); + + document.querySelectorAll('[data-theme-icon]').forEach(element => { + element.textContent = icon; + }); + }, + + init: () => { + window.formsTheme.set(window.formsTheme.get()); + } +}; + +document.addEventListener('DOMContentLoaded', () => { + window.formsTheme.init(); +}); + window.resizeInterop = { _formEntryActive: false, _formEntryStartX: 0, From 9753d4150841285bb5b19dddd34d0f549df81533 Mon Sep 17 00:00:00 2001 From: Maximum Code Date: Sat, 25 Apr 2026 08:12:25 -0700 Subject: [PATCH 04/10] Update website docs for v3.4 --- .../fulfillment-hub-sample-walkthrough.html | 270 ++++++++++++++++++ www/blog/index.html | 10 + www/changelog.html | 17 ++ www/docs/rest-api.html | 44 ++- www/js/csharpdb.bundle.js | 113 ++++++++ www/sitemap.xml | 8 +- 6 files changed, 450 insertions(+), 12 deletions(-) create mode 100644 www/blog/fulfillment-hub-sample-walkthrough.html diff --git a/www/blog/fulfillment-hub-sample-walkthrough.html b/www/blog/fulfillment-hub-sample-walkthrough.html new file mode 100644 index 00000000..9aa0bc01 --- /dev/null +++ b/www/blog/fulfillment-hub-sample-walkthrough.html @@ -0,0 +1,270 @@ + + + + + + + + Fulfillment Hub: A Guided CSharpDB Sample — Blog — CSharpDB + + + + + + + + + + + + + + +
+
+
+

Fulfillment Hub: A Guided CSharpDB Sample

+
+ Sample + April 25, 2026 + · 8 min read +
+ +
+

Most database samples are too small to teach much. A todo list can show inserts and queries, but it does not show what happens when a real application needs forms, reports, imports, exports, operational procedures, search, and semi-structured event data.

+

samples/fulfillment-hub is designed to be the opposite. It gives you a small warehouse and order-fulfillment system with one working database and enough moving parts to see how CSharpDB features fit together.

+ +
+ Goal: run the sample seeder, open the generated database in CSharpDB Admin, then walk through the same database as an operator using tables, views, forms, reports, procedures, saved queries, pipelines, collections, and full-text search. +
+ +

What the Sample Creates

+

Running the sample creates a fresh fulfillment-hub-demo.db database. The seeder loads more than SQL schema; it also stores the metadata-driven parts that Admin can run later.

+
    +
  • Relational tables for customers, products, inventory, orders, purchase orders, shipments, returns, carriers, suppliers, and audit events.
    • +
  • Views such as order_fulfillment_board, low_stock_watch, purchase_order_receiving_board, and shipment_manifest_report_source.
    • +
  • Stored procedures for allocation, receiving, shipment creation, returns, and operational stats.
    • +
  • Admin forms for order work, purchase-order receiving, and return intake.
    • +
  • Admin reports for the open order queue, low-stock watch, and shipment manifest.
    • +
  • Stored pipeline packages for CSV import, JSON import, and low-stock export.
    • +
  • Typed collections for scanner sessions and webhook archives.
    • +
  • A full-text index over operational playbooks.
    • +
+ +

Step 1: Run the Seeder

+

From the repository root, run the sample project:

+
+ 
dotnet run --project samples\fulfillment-hub\FulfillmentHubSample.csproj
+
+

Each run deletes and recreates the demo database in the build output folder:

+
+ 
samples\fulfillment-hub\bin\Debug\net10.0\fulfillment-hub-demo.db
+
+

The console output should show counts for forms, reports, procedures, saved queries, stored pipelines, pipeline runs, collections, and the full-text index. It also prints a few query results so you know the database was seeded successfully.

+ +

Step 2: Start Admin Against the Sample

+

CSharpDB Admin reads its local database from ConnectionStrings:CSharpDB. The easiest way to point Admin at the sample is to set the connection string for this terminal session and then run the Admin project.

+
+ 
$sampleDb = (Resolve-Path .\samples\fulfillment-hub\bin\Debug\net10.0\fulfillment-hub-demo.db).Path
+$env:ConnectionStrings__CSharpDB = "Data Source=$sampleDb"
+$env:CSharpDB__Transport = "direct"
+
+dotnet run --project src\CSharpDB.Admin\CSharpDB.Admin.csproj
+
+

The development launch profile uses these URLs:

+
    +
  • https://localhost:61816
    • +
  • http://localhost:61817
    • +
+

If those ports are already busy, run without the launch profile and choose a local URL:

+
+ 
dotnet run --no-launch-profile --project src\CSharpDB.Admin\CSharpDB.Admin.csproj -- --urls http://127.0.0.1:6190
+
+ +

Step 3: Confirm You Opened the Right Database

+

When Admin opens, check the title bar and Object Explorer. You should see the Fulfillment Hub database path and a populated object tree.

+

The useful signal is not just that tables exist. You should also see:

+
    +
  • Forms with Order Workbench, Purchase Order Receiving, and Return Intake.
    • +
  • Reports with Low Stock Watch, Open Order Queue, and Shipment Manifest.
    • +
  • Views with operational read models such as order_fulfillment_board and low_stock_watch.
    • +
  • Procedures with AllocateOrder, ReceivePurchaseOrder, CreateShipment, RecordReturn, and RefreshOperationalStats.
    • +
+ +

Step 4: Start With the Order Board

+

Open a query tab in Admin and run the main operational queue:

+
+ 
SELECT order_number, customer_name, warehouse_code, order_status, required_ship_date
+FROM order_fulfillment_board
+WHERE order_status IN ('released', 'allocated', 'picking')
+ORDER BY required_ship_date, priority_code DESC, order_number;
+
+

This is the first lesson in the sample: CSharpDB features are not isolated demos. The same order workflow is visible through a view, a saved query, a form, a report, and stored procedures.

+ +

Step 5: Use the Forms

+

Open Forms in the Object Explorer and start with Order Workbench. This form is bound to the orders table and includes child data for order lines.

+

Then open Purchase Order Receiving. That form is tied to the purchase-order model and helps you inspect the same receiving data that the procedure updates.

+

Finally, open Return Intake. It shows how reverse logistics can live beside outbound fulfillment without becoming a separate system.

+ +

Step 6: Run Operational Procedures

+

Procedures let you package multi-statement operational work. You can run them from SQL tabs with EXEC commands.

+
+ 
EXEC RefreshOperationalStats;
+
+EXEC ReceivePurchaseOrder purchaseOrderId=9001;
+
+EXEC AllocateOrder orderId=7005;
+
+EXEC CreateShipment orderId=7001, shipmentId=8101, shipmentNumber='SHP-8101', carrierId=2;
+
+

After each procedure, rerun the related view. This is where the sample starts to feel like a small operations system: receiving changes inventory, allocation changes order state, and shipment creation produces report-ready data.

+ +

Step 7: Open the Reports

+

The sample ships with three Admin reports. They are intentionally tied to the same operational story:

+
    +
  • Open Order Queue shows the work waiting on the floor.
    • +
  • Low Stock Watch shows shortage pressure by warehouse and SKU.
    • +
  • Shipment Manifest shows shipment-ready output from a report-shaped view.
    • +
+

Run CreateShipment, then open the shipment report. That connects the procedural workflow to the reporting surface.

+ +

Step 8: Inspect Pipelines

+

The seeder stores and runs three pipeline packages:

+
    +
  • supplier-receipts-import imports CSV receiving data.
    • +
  • marketplace-orders-import imports JSON marketplace orders.
    • +
  • low-stock-export exports the low-stock view to CSV.
    • +
+

In Admin, open the pipeline area and inspect the stored packages and run history. Then check the generated export file:

+
+ 
samples\fulfillment-hub\bin\Debug\net10.0\generated-output\low-stock-watch.csv
+
+ +

Step 9: Look at Collections and Full-Text Search

+

Fulfillment Hub also includes data that does not need to be forced into normal relational tables. The sample seeds two collections:

+
    +
  • scanner_sessions for handheld scanner state.
    • +
  • webhook_archive for external event payloads.
    • +
+

It also creates a full-text index called fts_ops_playbooks over the ops_playbooks table. That gives the sample a place for operational runbooks next to the live data.

+

The sample project demonstrates the full-text query through the engine API:

+
+ 
await using var db = await Database.OpenAsync("samples/fulfillment-hub/bin/Debug/net10.0/fulfillment-hub-demo.db");
+var hits = await db.SearchAsync("fts_ops_playbooks", "partial receipt");
+
+ +

How to Learn From the Sample

+

The best way to use Fulfillment Hub is to treat it like a live system, not a collection of files.

+
    +
  1. Run the seeder.
    2. +
  2. Open the generated database in Admin.
    3. +
  3. Run one query against a view.
    4. +
  4. Open the matching form or report.
    5. +
  5. Run a procedure that changes the workflow.
    6. +
  6. Reopen the view, form, and report to see what changed.
    7. +
+

That loop is the point of the sample. It shows how CSharpDB can hold the transactional model, the operational read models, the metadata-driven UI, the reporting layer, the integration pipelines, and supporting document-style data in one local database.

+ +
+ ← All posts + Next: CSharpDB Admin Launcher → +
+
+
+
+
+ + + + + diff --git a/www/blog/index.html b/www/blog/index.html index c0ff9e5b..505ee030 100644 --- a/www/blog/index.html +++ b/www/blog/index.html @@ -118,6 +118,16 @@

Blog

Tutorials, guides, and best practices for building with CSharpDB.

+ +
+ Sample + April 25, 2026 +
+

Fulfillment Hub: A Guided CSharpDB Sample

+

Seed a full warehouse and order-fulfillment database, then explore its tables, forms, reports, procedures, pipelines, collections, and full-text search through CSharpDB Admin.

+
Read article →
+ +
Guide diff --git a/www/changelog.html b/www/changelog.html index 7cd5ce77..f1a051e7 100644 --- a/www/changelog.html +++ b/www/changelog.html @@ -133,6 +133,23 @@

Changelog

Release announcements and project updates. For planned work, see the roadmap.

+
+
+ Release + April 2026 +
+

v3.4.0 — Admin Runtime, Fulfillment Hub Sample & Docs Polish

+

Focuses on making CSharpDB easier to explore locally: a richer Admin experience, a complete Fulfillment Hub sample database, a run-only forms web host, and clearer docs for launching and learning from the project:

+
    +
  • Fulfillment Hub sample — New end-to-end sample that seeds a working database with tables, views, stored procedures, forms, reports, pipelines, collections, and full-text search data. The walkthrough shows how to generate the database and open it through CSharpDB Admin.
    • +
  • Admin query workflow polish — Query autocomplete now handles LIMIT, UPDATE ... SET, WHERE column suggestions, and INSERT INTO ... columns ... VALUES flow more reliably.
    • +
  • Designer usability fixes — Form designer property fields now respect the current light/dark theme, and Query Designer views include resizable split bars so long SQL previews and result grids can be adjusted without losing context.
    • +
  • Forms-only web host — Added a run-only Forms Web project path for displaying saved forms from a CSharpDB database without exposing design mode. Static assets, shared styling, and the themed home screen now load correctly when started through the provided script.
    • +
  • Local Admin launcher guidance — New blog documentation explains how to create a small C# launcher executable that starts CSharpDB.Admin.exe, waits for the ASP.NET Core endpoint, and opens the default browser.
    • +
  • Website documentation polish — Added the Fulfillment Hub sample blog post, improved tutorial code color-coding across the static docs pages, refreshed release/PR notes, and kept the sitemap/blog index aligned with the new content.
    • +
+
+
Release diff --git a/www/docs/rest-api.html b/www/docs/rest-api.html index 20f5b0ba..02bf9655 100644 --- a/www/docs/rest-api.html +++ b/www/docs/rest-api.html @@ -6,10 +6,10 @@ REST API Reference — CSharpDB - + - + @@ -18,7 +18,7 @@ "@context": "https://schema.org", "@type": "TechArticle", "name": "REST API Reference", - "description": "CSharpDB REST API reference — all HTTP endpoints for database info, storage inspection, tables, rows, indexes, views, triggers, SQL execution, and procedures.", + "description": "CSharpDB daemon-hosted REST API reference — all HTTP endpoints for database info, storage inspection, tables, rows, indexes, views, triggers, SQL execution, and procedures.", "url": "https://csharpdb.com/docs/rest-api.html", "isPartOf": { "@type": "WebSite", @@ -33,8 +33,8 @@ window.currentPage = 'docs'; window.pagePathPrefix = '../'; window.pageConfig = { title: 'REST API Reference', - description: 'CSharpDB REST API reference — all HTTP endpoints for database info, storage inspection, tables, rows, indexes, views, triggers, SQL execution, and procedures.', - keywords: 'CSharpDB REST API, HTTP endpoints, tables, rows, indexes, views, triggers, SQL execution, procedures', + description: 'CSharpDB daemon-hosted REST API reference — all HTTP endpoints for database info, storage inspection, tables, rows, indexes, views, triggers, SQL execution, and procedures.', + keywords: 'CSharpDB REST API, CSharpDB Daemon, HTTP endpoints, tables, rows, indexes, views, triggers, SQL execution, procedures', canonicalPath: 'docs/rest-api.html', ogType: 'article', }; @@ -71,21 +71,42 @@

On This Page

CSharpDB REST API Reference

-

The CSharpDB REST API exposes the full database feature set over HTTP, enabling cross-language interoperability. Built with ASP.NET Core Minimal APIs, it includes OpenAPI documentation and an interactive Scalar UI.

+

The CSharpDB REST API is now hosted by CSharpDB.Daemon by default. The daemon exposes the existing HTTP /api surface and the gRPC service from one long-running process backed by the same warm database client.

+

The REST surface enables cross-language interoperability over JSON/HTTP. Built with ASP.NET Core Minimal APIs, it includes OpenAPI documentation and an interactive Scalar UI when the daemon runs in Development mode.

Running the API

- 
dotnet run --project src/CSharpDB.Api
-

The API starts on http://localhost:61818 (HTTP) and https://localhost:61819 (HTTPS).

-

Interactive documentation: Open http://localhost:61818/scalar/v1 in a browser to explore and test endpoints with the Scalar API explorer.

+

Run the combined daemon host from source:

+ 
dotnet run --project src/CSharpDB.Daemon/CSharpDB.Daemon.csproj
+

The daemon launch profile starts on https://localhost:49995 and http://localhost:49996. REST endpoints are available under /api, and gRPC endpoints are available from the same host.

+

Interactive documentation: In Development mode, open https://localhost:49995/scalar/v1 or http://localhost:49996/scalar/v1 in a browser to explore and test endpoints with the Scalar API explorer.

+

For a stable local HTTP port, set ASPNETCORE_URLS explicitly before starting the daemon:

+ 
$env:ConnectionStrings__CSharpDB = "Data Source=C:\data\sample.db"
+$env:ASPNETCORE_URLS = "http://localhost:5820"
+dotnet run --project src/CSharpDB.Daemon/CSharpDB.Daemon.csproj
+

With that override, the REST base URL is http://localhost:5820/api and Scalar is available at http://localhost:5820/scalar/v1.

Configuration

-

The default database path is configured in src/CSharpDB.Api/appsettings.json:

+

The current default database path and daemon host behavior are configured in src/CSharpDB.Daemon/appsettings.json:

{
   "ConnectionStrings": {
     "CSharpDB": "Data Source=csharpdb.db"
+  },
+  "CSharpDB": {
+    "Daemon": {
+      "EnableRestApi": true
+    },
+    "HostDatabase": {
+      "OpenMode": "HybridIncrementalDurable",
+      "ImplicitInsertExecutionMode": "ConcurrentWriteTransactions",
+      "UseWriteOptimizedPreset": true,
+      "HotTableNames": [],
+      "HotCollectionNames": []
+    }
   }
 }
+

CSharpDB:Daemon:EnableRestApi controls whether the daemon maps the REST /api surface. The default is true. Set it to false only when the daemon should expose gRPC without REST.

CORS is enabled for all origins by default (development convenience). JSON responses use camelCase naming and omit null values.

+

The standalone CSharpDB.Api project remains available for REST-only hosting, but the recommended remote host is CSharpDB.Daemon so REST and gRPC clients share the same warm database process.

Endpoints

All endpoints are prefixed with /api.

@@ -197,7 +218,7 @@

POST /api/maintenance/migrate-foreign-keys

  • validateOnly = true previews the migration without mutating schema or data.
  • backupDestinationPath is optional and is only used during apply mode.
    • -
  • Paths are resolved on the API host machine, not on the caller.
    • +
  • Paths are resolved on the daemon host machine, not on the caller.

Tables

@@ -525,6 +546,7 @@

Error Handling

See Also

  • Getting Started Tutorial — Engine API walkthrough
    • +
  • Multi-Writer gRPC Daemon — Daemon runtime model and remote-host guidance
  • Storage Architecture Deep Dive — How the engine works internally
  • CLI Reference — Interactive REPL commands
  • Sample Datasets — Ready-to-run SQL scripts
    • diff --git a/www/js/csharpdb.bundle.js b/www/js/csharpdb.bundle.js index 9fa41dc6..94362087 100644 --- a/www/js/csharpdb.bundle.js +++ b/www/js/csharpdb.bundle.js @@ -135,6 +135,118 @@ }); } + function escapeHtml(value) { + return value + .replace(/&/g, '&') + .replace(//g, '>') + .replace(/"/g, '"') + .replace(/'/g, '''); + } + + function inferCodeLanguage(code) { + const blockTitle = code.closest('.code-block')?.dataset.title?.toLowerCase() || ''; + const className = code.className.toLowerCase(); + const text = code.textContent || ''; + + if (blockTitle.includes('sql') || className.includes('language-sql')) return 'sql'; + if (blockTitle.includes('json') || className.includes('language-json') || /^\s*[{[]/.test(text)) return 'json'; + if (blockTitle.includes('powershell') || /\$env:|set-location|resolve-path/i.test(text)) return 'powershell'; + if (blockTitle.includes('bash') || className.includes('language-bash') || /(^|\n)\s*(dotnet|npm|node|cd|export)\b/.test(text)) return 'shell'; + if (blockTitle.includes('javascript') || className.includes('language-js') || /\b(import|const|let|function|await)\b[\s\S]*(from|require|console\.)/.test(text)) return 'javascript'; + if (blockTitle.includes('python') || className.includes('language-python') || /\b(def|from|import|with|None|True|False)\b/.test(text)) return 'python'; + if (blockTitle.includes('c#') || blockTitle.includes('program.cs') || className.includes('language-csharp')) return 'csharp'; + if (/\b(await|using|namespace|public|private|class|record|new|var|async|Task|Database|DbValue)\b/.test(text)) return 'csharp'; + + return 'text'; + } + + function tokenClass(token, language) { + const csharpKeywords = new Set([ + 'abstract', 'as', 'async', 'await', 'base', 'bool', 'break', 'byte', 'case', 'catch', + 'class', 'const', 'continue', 'decimal', 'default', 'do', 'double', 'else', 'enum', + 'false', 'finally', 'float', 'for', 'foreach', 'if', 'in', 'int', 'interface', + 'internal', 'is', 'long', 'namespace', 'new', 'null', 'object', 'out', 'override', + 'private', 'protected', 'public', 'readonly', 'record', 'return', 'sealed', 'short', + 'static', 'string', 'struct', 'switch', 'this', 'throw', 'true', 'try', 'uint', + 'using', 'var', 'void', 'while' + ]); + const jsKeywords = new Set([ + 'await', 'break', 'case', 'catch', 'class', 'const', 'continue', 'default', 'else', + 'export', 'false', 'finally', 'for', 'from', 'function', 'if', 'import', 'let', 'new', + 'null', 'return', 'throw', 'true', 'try', 'undefined', 'var', 'while' + ]); + const pythonKeywords = new Set([ + 'as', 'async', 'await', 'break', 'class', 'def', 'elif', 'else', 'except', 'False', + 'finally', 'for', 'from', 'if', 'import', 'in', 'is', 'None', 'return', 'True', 'try', + 'with', 'while' + ]); + const sqlKeywords = new Set([ + 'ADD', 'ALTER', 'AND', 'AS', 'ASC', 'BEGIN', 'BY', 'COMMIT', 'COUNT', 'CREATE', + 'DELETE', 'DESC', 'DISTINCT', 'DROP', 'EXEC', 'FROM', 'GROUP', 'INDEX', 'INSERT', + 'INTO', 'JOIN', 'KEY', 'LIMIT', 'NOT', 'NULL', 'ON', 'OR', 'ORDER', 'PRIMARY', + 'REAL', 'ROLLBACK', 'SELECT', 'SET', 'TABLE', 'TEXT', 'UPDATE', 'VALUES', 'VIEW', + 'WHERE' + ]); + const shellKeywords = new Set(['cd', 'dotnet', 'export', 'node', 'npm', 'python', 'set']); + const tokenUpper = token.toUpperCase(); + + if (/^(\/\/|\/\*)/.test(token)) return 'cm'; + if (token.startsWith('#') && (language === 'shell' || language === 'powershell' || language === 'python')) return 'cm'; + if (token.startsWith('--') && language === 'sql') return 'cm'; + if (/^(@?["']|"""|''')/.test(token)) return 'str'; + if (/^\d/.test(token)) return 'num'; + if (/^--[\w-]+$/.test(token) || /^\$[\w:]+$/.test(token)) return 'tp'; + + if (language === 'sql' && sqlKeywords.has(tokenUpper)) return 'kw'; + if (language === 'shell' && shellKeywords.has(token)) return 'kw'; + if (language === 'powershell' && (/^[A-Z][A-Za-z]+-[A-Za-z]+$/.test(token) || /^\$[\w:]+$/.test(token))) return 'kw'; + if (language === 'javascript' && jsKeywords.has(token)) return 'kw'; + if (language === 'python' && pythonKeywords.has(token)) return 'kw'; + if (language === 'json' && /^(true|false|null)$/i.test(token)) return 'kw'; + if (language === 'csharp' && csharpKeywords.has(token)) return 'kw'; + if (language === 'csharp' && /^[A-Z][A-Za-z0-9_]*$/.test(token)) return 'tp'; + + return ''; + } + + function highlightCodeText(source, language) { + if (language === 'text') return escapeHtml(source); + + const commentPatterns = ['\\/\\*[\\s\\S]*?\\*\\/']; + if (language === 'csharp' || language === 'javascript') commentPatterns.push('\\/\\/[^\\n]*'); + if (language === 'shell' || language === 'powershell' || language === 'python') commentPatterns.push('#[^\\n]*'); + if (language === 'sql') commentPatterns.push('--[^\\n]*'); + + const tokenPattern = new RegExp( + `(${commentPatterns.join('|')}|"""[\\s\\S]*?"""|'''[\\s\\S]*?'''|@?"(?:\\\\.|[^"\\\\])*"|'(?:\\\\.|[^'\\\\])*'|\\$[\\w:]+|--[\\w-]+|\\b\\d+(?:\\.\\d+)?\\b|\\b[A-Za-z_][A-Za-z0-9_]*\\b)`, + 'g' + ); + let html = ''; + let index = 0; + + source.replace(tokenPattern, (token, _unused, offset) => { + html += escapeHtml(source.slice(index, offset)); + const cls = tokenClass(token, language); + const escaped = escapeHtml(token); + html += cls ? `${escaped}` : escaped; + index = offset + token.length; + return token; + }); + + html += escapeHtml(source.slice(index)); + return html; + } + + function initCodeHighlighting() { + document.querySelectorAll('.doc-content pre code, .blog-post pre code').forEach(code => { + if (code.dataset.highlighted === 'true') return; + const language = inferCodeLanguage(code); + code.innerHTML = highlightCodeText(code.textContent || '', language); + code.dataset.highlighted = 'true'; + }); + } + document.addEventListener('DOMContentLoaded', () => { renderNav(); renderFooter(); @@ -142,6 +254,7 @@ initMobile(); initNavScroll(); initCopyButtons(); + initCodeHighlighting(); }); })(); diff --git a/www/sitemap.xml b/www/sitemap.xml index 9f4d4101..e42d3892 100644 --- a/www/sitemap.xml +++ b/www/sitemap.xml @@ -290,10 +290,16 @@ https://csharpdb.com/blog/ - 2026-04-23 + 2026-04-25 weekly 0.7 + + https://csharpdb.com/blog/fulfillment-hub-sample-walkthrough.html + 2026-04-25 + monthly + 0.6 + https://csharpdb.com/blog/csharpdb-admin-csharp-launcher.html 2026-04-23 From 16ba1e26ad7abed40403793f8a89162e9c59a6ae Mon Sep 17 00:00:00 2001 From: Maximum Code Date: Sat, 25 Apr 2026 08:19:03 -0700 Subject: [PATCH 05/10] Improve code formatting in architecture reference documentation --- www/architecture-reference.html | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/www/architecture-reference.html b/www/architecture-reference.html index 09ceb4b6..c0e1bbe7 100644 --- a/www/architecture-reference.html +++ b/www/architecture-reference.html @@ -95,7 +95,8 @@

    CSharpDB Architecture

    ships a reusable package-driven ETL pipeline runtime in CSharpDB.Pipelines that is reused by the client, API, CLI, and Admin surfaces.

    Layer Overview

    - 
    ┌────────────────────────────────────────────────────────────────────┐
+                
    
+                ┌────────────────────────────────────────────────────────────────────┐
                 │ Hosts / Applications                                               │
                 │ CSharpDB.Api   CSharpDB.Daemon   CSharpDB.Admin   CSharpDB.Cli     │
                 │ CSharpDB.Mcp                                                       │
@@ -229,7 +230,8 @@ 
    Page System
    
                 
                 
    Database File Format
    
                 
    The database is a sequence of 4096-byte pages. Page 0 contains the file header:
    
-                
    Offset  Size  Field
+                
    
+                Offset  Size  Field
                 ──────  ────  ─────
                 0       4     Magic bytes: "CSDB"
                 4       4     Format version (1)
@@ -243,7 +245,8 @@ 
    Database File Format
    
                 
    
                 
    Slotted Page Layout
    
                 
    Each B+tree page uses a slotted page format:
    
-                
    ┌───────────────────────────────────────────────────────────┐
+                
    
+                ┌───────────────────────────────────────────────────────────┐
                 │ Page Header (9 bytes)                                     │
                 │  [PageType:1] [CellCount:2] [ContentStart:2] [RightPtr:4] │
                 ├───────────────────────────────────────────────────────────┤
@@ -288,7 +291,8 @@ 
    Write-Ahead Log (WAL)
    
                 
                 
    CSharpDB uses a Write-Ahead Log for crash recovery and concurrent reader support. Modified pages are appended to a .wal file during commit, while the main .db file retains old data until checkpoint.
    
                 
    WAL File Format
    
-                
    ┌──────────────────────────────────────────────────────┐
+                
    
+                ┌──────────────────────────────────────────────────────┐
                 │ WAL Header (32 bytes)                                │
                 │  [magic:"CWAL"] [version:4] [pageSize:4]             │
                 │  [dbPageCount:4] [salt1:4] [salt2:4]                 │
@@ -301,7 +305,7 @@ 
    WAL File Format
    
                 ├──────────────────────────────────────────────────────┤
                 │ Frame 1 ...                                          │
                 ├──────────────────────────────────────────────────────┤
-                │ Frame N (commit frame: dbPageCount > 0)              │
+                │ Frame N (commit frame: dbPageCount > 0)           │
                 └──────────────────────────────────────────────────────┘
                 
    
                 
    Transaction Lifecycle (WAL Mode)
    
@@ -538,7 +542,8 @@ 
    Parsing Pipeline
    
                 
    
                 
    The tokenizer scans the input character by character, recognizing keywords (case-insensitive), identifiers, numeric literals (integer and real), string literals (single-quoted with '' escaping), operators, and punctuation.
    
                 
    The parser is a recursive descent parser. Each SQL statement type has its own parsing method. Expression parsing uses precedence climbing to correctly handle operator precedence:
    
-                
    Precedence (low to high):
+                
    
+                Precedence (low to high):
                   OR
                   AND
                   NOT (unary)
@@ -577,7 +582,8 @@ 
    Layer 4: Execution (CSharpDB
                 
                 
    Iterator Model
    
                 
    Query execution follows the Volcano/iterator model. Each operator implements IOperator:
    
-                
    public interface IOperator : IAsyncDisposable
+                
    
+                public interface IOperator : IAsyncDisposable
                 {
                     ColumnDefinition[] OutputSchema { get; }
                     ValueTask OpenAsync(CancellationToken ct = default);
@@ -1001,7 +1007,8 @@ 
    Layer 8: ADO.NET Provider (
                 
                 
    The ADO.NET provider allows CSharpDB to be used with the standard System.Data.Common APIs, making it compatible with ORMs and existing .NET data access code:
    
-                
    await using var conn = new CSharpDbConnection("Data Source=myapp.db");
+                
    
+                await using var conn = new CSharpDbConnection("Data Source=myapp.db");
                 await conn.OpenAsync();
                 
                 using var cmd = conn.CreateCommand();

From 827755629b4c60960561c25dfb13f03dee7a2054 Mon Sep 17 00:00:00 2001
From: Maximum Code 
Date: Sat, 25 Apr 2026 08:33:15 -0700
Subject: [PATCH 06/10] Remove outdated documentation for Entity Framework Core
 provider and multi-writer follow-up plan

---
 docs/entity-framework-core.md       |  97 ------
 docs/multi-writer-follow-up-plan.md | 439 ----------------------------
 2 files changed, 536 deletions(-)
 delete mode 100644 docs/entity-framework-core.md
 delete mode 100644 docs/multi-writer-follow-up-plan.md

diff --git a/docs/entity-framework-core.md b/docs/entity-framework-core.md
deleted file mode 100644
index 897e9540..00000000
--- a/docs/entity-framework-core.md
+++ /dev/null
@@ -1,97 +0,0 @@
-# Entity Framework Core 10 Provider
-
-`CSharpDB.EntityFrameworkCore` adds an embedded-only EF Core 10 provider on top of the existing ADO.NET layer in `CSharpDB.Data`.
-
-## Install
-
-For an app that consumes the package directly:
-
-```bash
-dotnet add package CSharpDB.EntityFrameworkCore
-dotnet add package Microsoft.EntityFrameworkCore.Design
-```
-
-Then configure your context with `UseCSharpDb(...)`:
-
-```csharp
-using CSharpDB.EntityFrameworkCore;
-using Microsoft.EntityFrameworkCore;
-
-public sealed class BloggingContext(string databasePath) : DbContext
-{
-    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
-        => optionsBuilder.UseCSharpDb($"Data Source={databasePath}");
-}
-```
-
-You can also pass an existing `CSharpDbConnection`:
-
-```csharp
-await using var connection = new CSharpDbConnection("Data Source=:memory:");
-await connection.OpenAsync();
-
-var options = new DbContextOptionsBuilder()
-    .UseCSharpDb(connection)
-    .Options;
-```
-
-Keep that connection open for the full lifetime of a private `:memory:` database.
-
-## Migrations
-
-For file-backed databases, the provider supports the normal EF Core design-time flow:
-
-```bash
-dotnet ef migrations add InitialCreate
-dotnet ef database update
-dotnet ef migrations script
-```
-
-`Database.Migrate()` and `EnsureCreated()` are both supported. Migrations use the standard `__EFMigrationsHistory` table plus a simple `__EFMigrationsLock` row to serialize concurrent migration runs across processes.
-
-## Supported Feature Matrix
-
-| Area | Supported in v1 | Notes |
-|------|------------------|-------|
-| Runtime provider | Yes | Embedded only |
-| File-backed databases | Yes | Primary supported mode |
-| Private `:memory:` runtime | Yes | Supply an open `CSharpDbConnection` |
-| `Database.Migrate()` | Yes | File-backed only |
-| `dotnet ef migrations add` | Yes | No separate design package required |
-| `dotnet ef database update` | Yes | File-backed only |
-| `dotnet ef migrations script` | Yes | Non-idempotent scripts |
-| `EnsureCreated()` | Yes | File-backed and private `:memory:` |
-| CRUD + change tracking | Yes | Includes affected-row concurrency checks |
-| Integer identity propagation | Yes | Single-column integer PKs |
-| Basic query subset | Yes | `Where`, ordering, pagination, scalar projections, `First`/`Single`, `Any`, `Count`, null checks, `Contains`, and simple navigation-loading joins |
-| Supported CLR types | Yes | `bool`, integral types, enums, `double`, `float`, `string`, `Guid`, `DateTime`, `DateTimeOffset`, `DateOnly`, `TimeOnly`, `byte[]` |
-| `decimal` | No | Add an explicit converter |
-| Schemas | No | Unsupported in runtime and migrations |
-| Defaults / computed / checks | No | Rejected early |
-| Rowversion | No | Unsupported |
-| Pooling | No | Provider rejects pooled connections |
-| Named shared-memory | No | Provider rejects `:memory:` |
-| Endpoint / daemon transports | No | Embedded-only provider in v1 |
-| Reverse engineering / scaffolding | No | Deferred |
-| Standalone FK alteration migrations | No | Only inline FKs in `CreateTable` |
-| Broad table-rebuild emulation | No | Unsupported operations fail explicitly |
-| Idempotent migration scripts | No | `dotnet ef migrations script --idempotent` is not supported in v1 |
-
-## DDL Surface
-
-The provider’s migrations SQL generator currently supports:
-
-- `CreateTable`
-- `DropTable`
-- `RenameTable`
-- `AddColumn`
-- `RenameColumn`
-- `DropColumn`
-- `CreateIndex`
-- `DropIndex`
-
-Foreign keys are supported only when emitted inline during `CreateTable`, and only for the current single-column shape supported by the engine.
-
-## Sample
-
-See [samples/efcore-provider](../samples/efcore-provider/README.md) for a minimal runnable app and design-time context factory.
diff --git a/docs/multi-writer-follow-up-plan.md b/docs/multi-writer-follow-up-plan.md
deleted file mode 100644
index 8e0de98c..00000000
--- a/docs/multi-writer-follow-up-plan.md
+++ /dev/null
@@ -1,439 +0,0 @@
-# Multi-Writer Follow-Up Plan
-
-This document tracks the work that remains after CSharpDB's initial multi-writer
-support. It is intentionally narrower than the original exploratory work: the
-engine already ships explicit `WriteTransaction` support, shared auto-commit
-non-insert isolation, and opt-in `ConcurrentWriteTransactions` for shared
-implicit inserts. What remains is the work needed to move from "initial support"
-to "multi-writer is broadly done and predictable."
-
----
-
-## Current State
-
-What is already shipped:
-
-- explicit multi-writer transactions through `BeginWriteTransactionAsync(...)`
-  and `RunWriteTransactionAsync(...)`
-- conflict detection and retry for isolated write-transaction attempts
-- shared auto-commit `UPDATE`, `DELETE`, and DDL routed through isolated
-  write-transaction state internally
-- opt-in `ImplicitInsertExecutionMode = ConcurrentWriteTransactions` for shared
-  auto-commit `INSERT`
-- retained-WAL, checkpoint, commit-fan-in, and insert-fan-in benchmark coverage
-
-What is still structurally true:
-
-- the physical WAL publish/flush path is still serialized at the storage
-  boundary
-- hot single-row insert workloads still do not show update-style fan-in
-- insert-side gains remain limited even when using explicit `WriteTransaction`
-- durable flush cost is still the dominant fixed cost once pending-commit
-  queuing works
-- user-facing gains are workload-dependent rather than a blanket "all writes
-  scale now"
-
-The April 10-13, 2026 benchmark closeout supports this reading:
-
-- low-conflict disjoint updates can coalesce durable flushes and benefit from
-  the current queue shape
-- hot insert loops still stay near `commitsPerFlush = 1.00`
-- the explicit write-transaction path is stable enough to ship, but the insert
-  path is still structurally limited
-
----
-
-## Definition Of Done
-
-We should only call multi-writer "done" when all of the following are true:
-
-1. Hot insert workloads no longer collapse to the legacy single-commit shape by
-   design alone.
-2. Auto-generated IDs and explicit IDs both behave predictably under concurrent
-   insert pressure without duplicate-key regressions or retry explosions.
-3. Shared auto-commit write traffic and explicit write transactions have
-   clearly documented and benchmarked performance envelopes.
-4. Checkpoint, WAL retention, and snapshot readers stay stable under sustained
-   write contention.
-5. Guardrail benchmarks can detect regressions in both low-conflict and
-   high-conflict multi-writer shapes.
-6. The storage and engine docs can make a clean statement about what kinds of
-   multi-writer workloads are expected to scale and which ones are still
-   intentionally serialized.
-
-Until then, the correct release language remains "initial multi-writer support."
-
----
-
-## Non-Goals
-
-This plan does not try to do the following in the same pass:
-
-- distributed replication or change-feed work
-- multi-process write coordination across separate daemon/engine processes
-- network transport optimizations unrelated to write-path contention
-- replacing the WAL durability model with a fundamentally different storage
-  engine
-
----
-
-## Workstreams
-
-The remaining work breaks into five concrete workstreams.
-
-### 1. Insert-Path Concurrency
-
-Problem:
-
-- concurrent inserts still collide structurally on row-id allocation,
-  uniqueness checks, and right-edge leaf growth
-- current fan-in data shows that changing the commit pipeline alone does not
-  unlock insert-side gains
-
-Primary goal:
-
-- make hot insert traffic conflict less before commit publication, not just
-  after it
-
-Likely design areas:
-
-- row-id reservation / allocation strategy for concurrent insert attempts
-- separation of "logical key assignment" from "physical leaf placement"
-- explicit handling of right-edge leaf split pressure
-- stronger rebase/retry support for insert intents after page split movement
-
-Likely code areas:
-
-- `src/CSharpDB.Engine/Database.cs`
-- `src/CSharpDB.Engine/WriteTransaction.cs`
-- `src/CSharpDB.Execution/QueryPlanner.cs`
-- `src/CSharpDB.Storage/Paging/Pager.cs`
-- `src/CSharpDB.Storage/Paging/LeafInsertRebaseHelper.cs`
-- `src/CSharpDB.Storage/Paging/InteriorInsertRebaseHelper.cs`
-
-Exit criteria:
-
-- insert fan-in benchmark no longer stays hard-pinned near
-  `commitsPerFlush = 1.00` for all hot-insert shapes
-- auto-id and explicit-id paths both remain correctness-stable
-- no duplicate-key regressions in the retry path
-
-### 2. Conflict Granularity And Rebase Stability
-
-Problem:
-
-- the system can detect conflicts, but insert and page-shape conflicts still
-  force more retry or structural fallback than we want
-
-Primary goal:
-
-- reduce unnecessary retries by making conflict detection more logical and less
-  page-incidental where correctness allows it
-
-Work items:
-
-- review current logical conflict key coverage
-- tighten the distinction between true logical conflicts and page-layout
-  movement
-- harden rebase helpers for parent/child page movement and late split
-  finalization
-- document which operations are intentionally page-structural versus logically
-  mergeable
-
-Likely code areas:
-
-- `src/CSharpDB.Storage/Transactions/LogicalConflictKey.cs`
-- `src/CSharpDB.Storage/Transactions/TransactionCoordinator.cs`
-- `src/CSharpDB.Storage/Transactions/PagerTransactionState.cs`
-- `src/CSharpDB.Storage/Paging/LeafInsertRebaseHelper.cs`
-- `src/CSharpDB.Storage/Paging/InteriorInsertRebaseHelper.cs`
-
-Exit criteria:
-
-- lower retry counts for disjoint-key insert/update scenarios
-- no new corruption or phantom-success paths under `DatabaseConcurrencyTests`
-- conflict behavior explainable in docs and diagnostics
-
-### 3. Commit Pipeline And Durable Flush Sharing
-
-Problem:
-
-- once writes reach the pending-commit queue, durable flush still dominates
-  cost
-- the queue now helps low-conflict non-insert traffic, but the overall commit
-  path still needs clearer stage boundaries
-
-Primary goal:
-
-- keep improving the publish/finalize pipeline without weakening durability or
-  correctness
-
-Work items:
-
-- continue separating WAL append, publish, and durable finalization stages
-- verify whether more publish-side batching is possible without reordering
-  committed visibility incorrectly
-- measure whether pending-commit queue depth can increase safely for the
-  non-insert path
-- validate whether `UseDurableGroupCommit(...)` needs smarter defaults or
-  remains strictly opt-in
-
-Likely code areas:
-
-- `src/CSharpDB.Storage/Wal/WriteAheadLog.cs`
-- `src/CSharpDB.Storage/Wal/IWriteAheadLog.cs`
-- `src/CSharpDB.Storage/Paging/Pager.cs`
-- `src/CSharpDB.Storage/Transactions/TransactionCoordinator.cs`
-- `src/CSharpDB.Storage/StorageEngine/DurableGroupCommitOptions.cs`
-
-Exit criteria:
-
-- disjoint-update commit fan-in remains stable under reruns
-- no regression in single-writer durability numbers beyond accepted thresholds
-- commit stage diagnostics remain understandable and monotonic
-
-### 4. Reader / Checkpoint / Retained-WAL Interaction
-
-Problem:
-
-- multi-writer throughput is not just a write path issue; checkpoint retention,
-  reader snapshots, and retained WAL can become secondary bottlenecks or hide
-  regressions
-
-Primary goal:
-
-- ensure that the multi-writer path stays stable when readers and checkpoints
-  coexist with sustained write traffic
-
-Work items:
-
-- revalidate retained-WAL compaction behavior
-- keep checkpoint finalization behavior explicit when snapshots still reference
-  WAL frames
-- stress test mixed read/write scenarios with hybrid mode enabled
-- document acceptable retained-WAL growth and backpressure behavior
-
-Likely code areas:
-
-- `src/CSharpDB.Storage/Paging/Pager.cs`
-- `src/CSharpDB.Storage/Wal/WriteAheadLog.cs`
-- `tests/CSharpDB.Tests/WalTests.cs`
-- `tests/CSharpDB.Tests/DatabaseConcurrencyTests.cs`
-
-Exit criteria:
-
-- no large manual-checkpoint tails after the writer blocker is released
-- mixed read/write tests stay deterministic
-- retained-WAL behavior remains bounded and diagnosable
-
-### 5. Benchmarks, Guardrails, And Release Criteria
-
-Problem:
-
-- the engine now has enough write-path branches that correctness alone is not a
-  sufficient release signal
-
-Primary goal:
-
-- turn the current benchmark knowledge into a durable release contract
-
-Work items:
-
-- keep `WriteTransactionDiagnosticsBenchmark` as the explicit transaction
-  baseline
-- keep `CommitFanInDiagnosticsBenchmark` for shared non-insert queue behavior
-- keep `InsertFanInDiagnosticsBenchmark` as the insert-side structural truth
-  source
-- keep `ConcurrentDurableWriteBenchmark` for shared-database writer contention
-- decide which rows are release-blocking versus informative only
-- refresh perf thresholds only after same-runner reruns
-
-Primary files:
-
-- `tests/CSharpDB.Benchmarks/README.md`
-- `tests/CSharpDB.Benchmarks/perf-thresholds.json`
-- benchmark classes under `tests/CSharpDB.Benchmarks/Macro`
-
-Exit criteria:
-
-- release guardrails match the actual shipped write contract
-- stale thresholds do not block release for the wrong reasons
-- docs can point to stable benchmark sources for each supported workload shape
-
----
-
-## Recommended Execution Order
-
-The clean order is:
-
-1. Insert-path concurrency design
-2. Conflict granularity and rebase hardening
-3. Commit pipeline refinement
-4. Mixed read/write + checkpoint validation
-5. Benchmark threshold and docs refresh
-
-Reasoning:
-
-- the insert path is still the main structural gap
-- commit-pipeline tuning without insert-side redesign will keep hitting the
-  same ceiling
-- benchmark work should validate the new shape, not guess it in advance
-
----
-
-## Detailed Phase Plan
-
-### Phase A: Insert Architecture Design
-
-Deliverables:
-
-- a written design for concurrent row-id reservation and insert intent handling
-- a clear statement about whether the chosen path is optimistic reservation,
-  preallocated ranges, or another model
-- a correctness matrix for explicit-id versus auto-id inserts
-
-Implementation notes:
-
-- keep the design compatible with current WAL durability and page layout
-- do not weaken uniqueness enforcement just to reduce retries
-- prefer explicit diagnostics over silent fallback to the legacy path
-
-Tests to add or strengthen:
-
-- concurrent auto-id insert stress with retries and reopen validation
-- concurrent explicit-id insert stress on disjoint keys
-- split-heavy insert scenarios that force leaf and parent movement
-
-Phase exit:
-
-- design approved and minimally implemented behind the current public API
-
-### Phase B: Insert Rebase / Retry Implementation
-
-Deliverables:
-
-- implemented row-id / insert-intent improvements
-- hardened insert rebase helpers
-- conflict/retry telemetry for the new path
-
-Tests to add or strengthen:
-
-- `DatabaseConcurrencyTests` coverage for repeated page splits
-- WAL recovery coverage for partially progressed concurrent insert attempts
-- long-running insert loops with seeded and unseeded tables
-
-Benchmarks to rerun:
-
-- `WriteTransactionDiagnosticsBenchmark`
-- `InsertFanInDiagnosticsBenchmark`
-
-Phase exit:
-
-- insert-side numbers materially improve or the design is clearly proven not to
-  help and is revised before moving on
-
-### Phase C: Commit Pipeline Consolidation
-
-Deliverables:
-
-- clearer separation of append, publish, and durable finalize stages
-- stable pending-commit queue behavior under multi-writer non-insert traffic
-- updated diagnostics fields if stage timing changes
-
-Tests to add or strengthen:
-
-- queue-depth and flush-sharing assertions
-- cancellation and disposal behavior while commits are pending
-- crash-recovery coverage after queued but not yet finalized work
-
-Benchmarks to rerun:
-
-- `CommitFanInDiagnosticsBenchmark`
-- `ConcurrentDurableWriteBenchmark`
-- targeted durable-write comparisons against single-writer baselines
-
-Phase exit:
-
-- queue behavior is stable and does not regress single-writer durability
-
-### Phase D: Mixed Workload Hardening
-
-Deliverables:
-
-- stable mixed read/write/checkpoint behavior under the new path
-- retained-WAL and checkpoint guidance updated if necessary
-
-Tests to add or strengthen:
-
-- concurrent readers plus insert-heavy writers
-- hybrid incremental-durable mixed workloads
-- checkpoint-retention and post-checkpoint recovery scenarios
-
-Benchmarks to rerun:
-
-- `checkpoint-retention-diagnostics`
-- relevant hybrid storage and concurrent read suites
-
-Phase exit:
-
-- no regression in mixed workload behavior or retained-WAL safety
-
-### Phase E: Release Contract Refresh
-
-Deliverables:
-
-- benchmark README refresh
-- perf threshold refresh where justified
-- storage / engine / roadmap docs updated to the new supported boundary
-
-Release gate:
-
-- correctness tests green
-- focused multi-writer benchmarks rerun on the same runner
-- threshold updates justified by fresh captured artifacts, not anecdotes
-
----
-
-## Testing Matrix
-
-Every major multi-writer follow-up change should be validated against this
-matrix:
-
-| Area | Minimum Validation |
-|------|--------------------|
-| Correctness | `tests/CSharpDB.Tests/DatabaseConcurrencyTests.cs` plus WAL recovery coverage |
-| Storage safety | `tests/CSharpDB.Tests/WalTests.cs` and checkpoint-related tests |
-| Explicit multi-writer API | `WriteTransactionDiagnosticsBenchmark` and targeted transaction tests |
-| Shared auto-commit non-insert | `CommitFanInDiagnosticsBenchmark` |
-| Shared auto-commit insert | `InsertFanInDiagnosticsBenchmark` |
-| Shared durable contention | `ConcurrentDurableWriteBenchmark` |
-| Release guardrails | `tests/CSharpDB.Benchmarks/perf-thresholds.json` compare run on the canonical perf runner |
-
----
-
-## Open Questions
-
-These questions should be answered before we claim broader completion:
-
-1. Should auto-generated row IDs reserve ranges per writer attempt, or should
-   they remain globally coordinated but decoupled from page placement?
-2. Can right-edge insert pressure be reduced enough with reservation/rebase
-   alone, or does the tree need a more explicit append-friendly shape?
-3. Is there a correctness-safe way to let more insert work reach the pending
-   commit queue before leaf placement is finalized?
-4. Which benchmark rows should become release-blocking for insert-side behavior,
-   given that some shapes are intentionally still serialized today?
-5. Do we want a separate public knob for insert-path strategy, or should this
-   remain internal behind `ImplicitInsertExecutionMode`?
-
----
-
-## Recommended Short Version
-
-If we want the shortest accurate summary for future planning:
-
-- initial multi-writer support is done
-- low-conflict non-insert fan-in is meaningfully better
-- hot insert fan-in is still the main unfinished structural problem
-- the next serious phase should focus on row-id reservation, insert intent
-  rebasing, and right-edge insert pressure before doing more queue tuning

From a7f7002616df5861f72d74f87f83f4452b24f94c Mon Sep 17 00:00:00 2001
From: Maximum Code 
Date: Sat, 25 Apr 2026 08:33:47 -0700
Subject: [PATCH 07/10] fix links

---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index f377d12a..042acf19 100644
--- a/README.md
+++ b/README.md
@@ -207,7 +207,7 @@ The native library exports 20 C functions. See the [Native Library Reference](ht
 | | |
 |---|---|
 | [Getting Started](https://csharpdb.com/getting-started.html) | Step-by-step walkthrough |
-| [Architecture Guide](https://csharpdb.com/docs/architecture.html) | Engine design deep dive |
+| [Architecture Guide](https://csharpdb.com/architecture.html) | Engine design deep dive |
 | [Tools & Ecosystem](https://csharpdb.com/docs/ecosystem.html) | APIs, hosts, designers, and integrations |
 | [EF Core Provider](docs/entity-framework-core.md) | Embedded EF Core 10 provider guide |
 | [Admin UI Guide](https://csharpdb.com/docs/admin-ui.html) | Querying, schema, pipelines, forms, reports, and storage |

From cdf5dffadfc664b90102277afc621acd67bd6bfa Mon Sep 17 00:00:00 2001
From: Maximum Code 
Date: Sat, 25 Apr 2026 10:12:54 -0700
Subject: [PATCH 08/10] update roadmap

---
 www/roadmap.html | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/www/roadmap.html b/www/roadmap.html
index 90f4863a..1b41db32 100644
--- a/www/roadmap.html
+++ b/www/roadmap.html
@@ -131,7 +131,7 @@
         
    
             
    
                 
    Roadmap
    
-                
    Planned direction for CSharpDB — organized by timeframe and priority. Reflects the current v3.0.0 state.
    
+                
    Planned direction for CSharpDB — organized by timeframe and priority. Reflects the current v3.4.0 state.
    
                 
    
                     Need the full source guide? The original long-form markdown version is preserved as Roadmap Source Reference.
                 
    
@@ -314,9 +314,9 @@ 
    Foreign Key Constraints
    
                         
    
                             
    
                                 
    Remote Host Consolidation
    
-                                Planned
+                                Done
                             
    
-                            
    Fold REST/HTTP into CSharpDB.Daemon so one server host serves REST, gRPC, and future transports from a shared warm Database instance.
    
+                            
    CSharpDB.Daemon now hosts the existing REST/HTTP /api surface and gRPC from one long-running process backed by the same warm daemon-hosted client. Standalone CSharpDB.Api remains supported for REST-only hosting.
    
                         
    
                         
    
                             
    
@@ -328,16 +328,16 @@ 
    Remote Host Security
    
                         
    
                             
    
                                 
    Daemon Service Packaging
    
-                                Planned
+                                Done
                             
    
-                            
    Package CSharpDB.Daemon as a persistent background service across systemd, Windows Service, and launchd.
    
+                            
    CSharpDB.Daemon can be packaged as a persistent background service across systemd, Windows Service, and launchd.
    
                         
    
                         
    
                             
    
                                 
    Cross-Platform Distribution
    
-                                Planned
+                                In Progress
                             
    
-                            
    dotnet tool, self-contained binaries, Docker, Homebrew, winget, and install scripts.
    
+                            
    Self-contained daemon archives and install scripts ship for Windows, Linux, and macOS; dotnet tool, Docker, Homebrew, and winget distribution remain future work.
    
                         
    
                         
    
                             
    
@@ -488,8 +488,8 @@ 
    Current Limitations
    
                             IndexesEquality lookups support current INTEGER/TEXT indexes, but ordered range-scan pushdown is still limited to single-column INTEGER index paths
                             RowIdLegacy table schemas without persisted high-water metadata may pay a one-time key scan on first insert
                             CollectionsFindByIndexAsync supports declared field-equality lookups; FindByPathAsync and FindByPathRangeAsync support path-based queries on indexed paths; FindAsync remains a full scan for unindexed predicates
-                            NetworkingRemote access split between CSharpDB.Api (HTTP) and CSharpDB.Daemon (gRPC); host consolidation plus named pipes remain planned
-                            SecurityNo built-in authentication/authorization or TLS/mTLS; relies on external network controls or front-end TLS termination
+                            NetworkingCSharpDB.Daemon now hosts both REST and gRPC from one process; named pipes remain reserved but are not implemented end to end today
+                            SecurityRemote HTTP and gRPC deployment still rely on external network controls or front-end TLS termination; built-in authentication, authorization, and TLS/mTLS support are still planned
                             CollationDefault semantics remain ordinal, but opt-in BINARY, NOCASE, NOCASE_AI, and ICU:<locale> collation are implemented for SQL and collection indexes; dedicated ordered SQL text index optimization remains planned
                             ConcurrencyPhysical WAL commit path is still serialized at the storage boundary. Initial multi-writer support is shipped, but observed gains depend on conflict shape and whether shared auto-commit INSERT is left on the default serialized path
                             StorageNo page-level compression
@@ -541,6 +541,8 @@ 
    Completed Milestones
    
                         
    ✓ ReplaceAsync for index stores
    
                         
    ✓ Maintenance report, REINDEX, and VACUUM flows across client, CLI, API, and Admin UI
    
                         
    ✓ Dedicated gRPC daemon host
    
+                        
    ✓ Remote host consolidation in CSharpDB.Daemon, with REST /api and gRPC sharing one warm hosted database client
    
+                        
    ✓ Daemon service packaging with self-contained archives and service install assets
    
                         
    ✓ Storage tuning presets, bounded WAL read caching, memory-mapped reads, and sliced background checkpointing
    
                         
    ✓ SQL executor/read-path fast paths for compact projections, broader join/index coverage, and correlated subquery filters
    
                         
    ✓ REST API with 34+ endpoints and OpenAPI/Scalar documentation
    

From 58329f08a18ba6fddb044b86715882f4ef13174a Mon Sep 17 00:00:00 2001
From: Maximum Code 
Date: Sat, 25 Apr 2026 12:47:22 -0700
Subject: [PATCH 09/10] add ignore mcp file

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 26c4c4f2..c0fdcef5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,6 +12,7 @@ tmp/
 *.userosscache
 *.sln.docstates
 .dotnet-cli/
+.playwright-mcp/
 
 ## VS / Rider IDE
 .vs/

From 42ab26ee913876c98617a98d36e953643d4e99f9 Mon Sep 17 00:00:00 2001
From: Maximum Code 
Date: Sat, 25 Apr 2026 12:49:43 -0700
Subject: [PATCH 10/10] Update v3.4 PR notes

---
 docs/releases/v3.4.0-pr-notes.md | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/docs/releases/v3.4.0-pr-notes.md b/docs/releases/v3.4.0-pr-notes.md
index 4f693b7a..32e3ed96 100644
--- a/docs/releases/v3.4.0-pr-notes.md
+++ b/docs/releases/v3.4.0-pr-notes.md
@@ -28,6 +28,11 @@ the SQL query tab has a visible resizable splitter between editor and results,
 and the visual query designer has a matching splitter for the generated SQL
 preview versus results.
 
+The final website pass also adds the Fulfillment Hub sample walkthrough blog
+post, applies consistent tutorial code color-coding across the static docs,
+updates the REST API reference to reflect daemon-hosted REST/gRPC, and refreshes
+the v3.4 changelog plus roadmap entries.
+
 ## Type of Change
 
 - [x] Bug fix
@@ -61,7 +66,7 @@ work. Included work in this PR:
 ## Testing
 
 - [ ] `dotnet build CSharpDB.slnx`
-- [ ] Relevant tests executed
+- [x] Relevant tests executed
 - [ ] Failure-path tests executed (if applicable: cancellation, invalid/unsupported inputs, non-`DbException` paths)
 - [x] Manual verification performed (if applicable)
 
@@ -99,6 +104,21 @@ Validation performed for this PR:
 - `dotnet build src\CSharpDB.Admin\CSharpDB.Admin.csproj -p:BaseOutputPath=C:\Users\maxim\source\Code\CSharpDB\artifacts\verify\`
   completed successfully after the query tab and visual designer splitter
   changes
+- Sequential unit-test run completed with `1644` passed, `0` failed, and `0`
+  skipped across:
+  - `CSharpDB.Admin.Forms.Tests`
+  - `CSharpDB.Admin.Reports.Tests`
+  - `CSharpDB.Api.Tests`
+  - `CSharpDB.Cli.Tests`
+  - `CSharpDB.Daemon.Tests`
+  - `CSharpDB.Data.Tests`
+  - `CSharpDB.EntityFrameworkCore.Tests`
+  - `CSharpDB.Pipelines.Tests`
+  - `CSharpDB.Tests`
+- `node --check www\js\csharpdb.bundle.js` passed after adding the static docs
+  code highlighter
+- browser smoke verification confirmed all tutorial code blocks are processed
+  and the REST API reference renders with daemon-hosted REST wording
 - `dotnet run --project src\CSharpDB.Admin.Forms.Web\CSharpDB.Admin.Forms.Web.csproj -- --urls http://127.0.0.1:5095 --CSharpDB:DataSource=`
   started successfully
 - HTTP verification against the new forms host confirmed:
@@ -134,3 +154,6 @@ Validation performed for this PR:
 - The latest Admin UI fixes are narrow and local: one theme/readability fix in
   the form designer property inspector, one splitter for SQL mode, and one
   splitter for visual designer mode.
+- The static website updates are documentation-only, except for the shared
+  client-side highlighter in `www/js/csharpdb.bundle.js`; it uses the existing
+  site token classes and does not add a new third-party dependency.