Both CSharpDB and SQLite are excellent embedded databases for .NET, and "which one is faster?" is the wrong question. The right question is: for each workload, which API on which engine gets you the best number? Sometimes that's SQLite through ADO.NET. Sometimes it's CSharpDB through the direct engine API. Sometimes the gap between the two engines is smaller than the gap between a good API choice and a bad one on either side.
This post walks through four paired benchmarks on .NET 10 — bulk inserts, point lookups, concurrent writers, and EF Core — and shows the compilable code that produced each number. Everything in this post runs end-to-end with dotnet run -c Release.
docs/query-and-durable-write-performance/csharpdb-vs-sqlite-benchmarking-blog.md. For the shorter curated page, see CSharpDB vs SQLite: A Benchmark-Driven Guide.CSharpDB Versus SQLite: How We Compare Them Fairly, and How To Get the Most Out of CSharpDB
+If you want to compare an embedded database against SQLite, it is very easy to + publish a misleading chart without meaning to.
+The trap is always the same: one side gets measured at the engine layer, the + other side gets measured through a provider or ORM, durability settings do not + match, batching rules differ, and the final table looks precise while answering + the wrong question.
+This post is the practical version of how we compare CSharpDB and SQLite in + this repository. It also doubles as a guide for getting the best insert + performance out of CSharpDB when you are using it in a real application.
+The short version is:
+-
+
- Compare engine to engine, provider to provider, and ORM to ORM. +
- Keep durability contracts aligned. +
- Reuse prepared work on both sides. +
- Treat single-writer bulk ingest and concurrent writers as different + workloads. +
- Use the CSharpDB surfaces that avoid per-row SQL and planner overhead when + you care about write throughput. +
Why Fair Comparisons Are Hard
+The phrase "CSharpDB versus SQLite" sounds like one benchmark, but in practice + it is at least four different comparisons:
+-
+
- CSharpDB engine API versus SQLite engine API. +
- CSharpDB ADO.NET provider versus SQLite ADO.NET provider. +
- CSharpDB EF Core provider versus SQLite EF Core provider. +
- Single-writer durable bulk ingest versus concurrent writer contention. +
Those are not interchangeable.
+If one result uses the raw engine on one side and ADO.NET on the other, the + chart is already mixing surfaces. If one result uses file-backed durable WAL + and the other uses shared memory, it is no longer a storage-engine comparison. + If one side prepares a command once and reuses it while the other reparses SQL + for every row, the result mostly measures statement setup rather than insert + throughput.
+That is why the CSharpDB benchmark suite splits these questions on purpose.
+The Comparison Rules We Use
+The repo follows a few rules consistently.
+Rule 1: Match the Layer
+We keep these pairings separate:
+-
+
- Engine versus engine. +
- ADO.NET versus ADO.NET. +
- EF Core versus EF Core. +
That means:
+-
+
- If we compare CSharpDB directly through
DatabaseandInsertBatch, we + compare it against SQLite's native C API rather than against +Microsoft.Data.Sqlite.
+ - If we compare
CSharpDB.Data, we compare it against +Microsoft.Data.Sqlite.
+ - If we compare the EF Core provider, we compare it against EF Core on SQLite. +
Rule 2: Match the Durability Contract
+For the primary durable write comparison, both sides are file-backed and + durable.
+For the matched SQLite baseline in + SqliteComparisonBenchmark.cs, + SQLite uses:
+-
+
journal_mode=WAL
+ synchronous=FULL
+ - explicit transaction batching +
- prepared statement reuse +
For the corresponding CSharpDB bulk path in + DurableSqlBatchingBenchmark.cs, + the comparison uses the engine's durable file-backed path with the write-heavy + storage preset and the same logical batch shape.
+Rule 3: Match the Statement Shape
+SQLite absolutely does support the classic "prepare once, bind many times"
+ workflow. In the native API that is sqlite3_prepare_v2, followed by repeated
+ binds and sqlite3_step / reset cycles. In ADO.NET the equivalent is
+ DbCommand.Prepare() plus reusing parameters.
CSharpDB has two comparable fast paths, depending on the layer:
+-
+
- At the engine layer, use
Database.PrepareInsertBatch(...).
+ - At the ADO.NET layer, use
DbCommand.Prepare()throughCSharpDB.Data.
+
If you compare SQLite with prepared reuse against CSharpDB with one fresh SQL + string per row, you are not comparing the engines fairly.
+Rule 4: Keep Single-Writer and Multi-Writer Results Separate
+Single-writer durable ingest answers one question:
+How fast can each engine append committed rows when batching and durability are + held constant?
+Concurrent writers answer a different question:
+How does each engine behave when several local writers are inserting at the + same time against the same database?
+Those are different workloads with different limits. They should not be merged + into one headline number.
+What the Current Durable Bulk Numbers Say
+The cleanest apples-to-apples insert comparison in this repo is still the + single-writer durable four-column bulk row:
+-
+
- schema:
+
id INTEGER PRIMARY KEY, value INTEGER, text_col TEXT, category TEXT
+ - monotonic primary keys +
- explicit transaction batching +
- file-backed durable mode +
- median-of-3 reruns on the same runner +
As of April 21, 2026 (promoted from the release-core run with
+ PASS=185, WARN=0, SKIP=0, FAIL=0), the latest matched rerun on this machine
+ produced:
| Scenario | +Rows/sec | +P50 | +P99 | +
|---|---|---|---|
CSharpDB InsertBatch B1000 |
+ 204,028 |
+ 4.1034 ms | +9.5599 ms | +
SQLite prepared bulk B1000 |
+ 192,059 |
+ 4.7479 ms | +18.1078 ms | +
CSharpDB InsertBatch B10000 |
+ 798,254 |
+ 8.7019 ms | +119.0664 ms | +
SQLite prepared bulk B10000 |
+ 539,562 |
+ 16.6275 ms | +43.3034 ms | +
That means, on this runner:
+-
+
- CSharpDB is about
106.2%of SQLite atB1000.
+ - CSharpDB is about
147.9%of SQLite atB10000.
+
Source artifacts:
+-
+
- durable-sql-batching-20260421-214227-median-of-3.csv +
- sqlite-compare-20260421-222824-median-of-3.csv +
- Full scorecard: tests/CSharpDB.Benchmarks/SQLITE_COMPARISON.md +
The important part is not just that CSharpDB is ahead on this matched row. The + important part is what that row actually proves:
+-
+
- It proves the primary durable monotonic bulk path is now competitive. +
- It does not prove every insert shape is faster. +
- It does not erase the cost of secondary indexes. +
- It does not erase the random-key locality cliff. +
- It does not answer hot right-edge multi-writer contention. +
That distinction matters if you want the public write-up to stay honest.
+The Three Real Benchmark Surfaces
+The benchmark suite now lets us talk about three comparison layers clearly.
+1. Direct Engine Versus SQLite C API
+This comparison lives in + ConcurrentSqliteCApiComparisonBenchmark.cs.
+On the CSharpDB side:
+-
+
- one shared
Databaseinstance is opened for the scenario
+ - one writer task is created per logical writer +
- each writer loops for the measurement window +
- each loop performs one insert commit +
- the surface is either raw
Database.ExecuteAsync(sql)or a reused + one-rowInsertBatch
+ - concurrent insert routing can opt into
+
ImplicitInsertExecutionMode.ConcurrentWriteTransactions
+
On the SQLite side:
+-
+
- one native SQLite connection handle is opened per writer +
- one prepared native insert statement is created per writer +
- each loop rebinds values, calls
sqlite3_step, then resets the statement
+ - SQLite runs with
WAL,FULL, and abusy_timeout
+
This is the right layer for answering engine-versus-engine concurrency + questions.
+2. ADO.NET Versus ADO.NET
+This comparison lives in + ConcurrentAdoNetComparisonBenchmark.cs + and + StrictInsertComparisonBenchmark.cs.
+The provider rules are intentionally symmetrical:
+-
+
- one connection per writer +
- one prepared command per writer +
- one
ExecuteNonQueryAsync()call per insert
+
The strict single-writer comparison also covers:
+-
+
- raw SQL single inserts +
- prepared single inserts +
- raw SQL batched inserts +
- prepared batched inserts +
- file-backed CSharpDB ADO.NET opened with
+
Storage Preset=WriteOptimizedandEmbedded Open Mode=Direct
+ - file-backed SQLite ADO.NET with
journal_mode=WALandsynchronous=FULL
+
This is the right layer for answering provider overhead questions. It is not + the right layer for making storage-engine claims unless the durability contract + also matches.
+3. EF Core Versus EF Core
+This comparison lives in + EfCoreComparisonBenchmark.cs.
+Today it covers:
+-
+
--efcore-compare: single-row and100-rowSaveChangesAsync()with one + open connection held for the timed run
+ --efcore-compare-hybrid-shared-connection: the same insert shapes with + one externally-owned open connection reused across short-livedDbContext+ instances
+ --efcore-compare-auto-open-close: the same insert shapes with EF-managed + auto-open/close left in the measurement
+ - CSharpDB EF Core provider versus SQLite EF Core provider +
- CSharpDB EF Core configured with
UseStoragePreset(WriteOptimized)and +UseEmbeddedOpenMode(Direct)
+
It does not currently provide a dedicated concurrent EF Core benchmark. So if + you are writing a blog post, the fair statement is:
+We have an EF Core comparison surface, but the concurrent comparison story is + currently strongest at the engine and ADO.NET layers.
+How Concurrent Writer Tests Differ Between the Two Databases
+This is usually where comparison posts become hand-wavy, so it is worth being + explicit.
+For CSharpDB concurrent engine tests:
+-
+
- writers share one in-process engine +
- the engine decides whether inserts stay serialized or route through isolated + write transactions +
- disjoint explicit-key workloads can benefit from the concurrent insert path +
- hot right-edge workloads still behave very differently from disjoint-key + workloads +
For SQLite concurrent engine tests:
+-
+
- each writer has its own native handle to the same database file +
- every writer still competes inside SQLite's WAL write rules +
- prepared statements are reused, but the engine remains effectively + single-writer at the durable file boundary +
So the concurrency comparison is not "who has more threads." It is "how does + each engine's write architecture behave when several local writers are active."
+How To Get the Best Performance Out of CSharpDB
+This is the part that matters if you are not just comparing benchmarks but also + trying to ship a fast application.
+Start With the Engine Fast Path
+If you are using the engine directly, the highest-value insert surface is
+ PrepareInsertBatch(...), not one dynamically built SQL string per row.
using CSharpDB.Engine;
+ using CSharpDB.Primitives;
+
+ var options = new DatabaseOptions()
+ .ConfigureStorageEngine(builder => builder.UseWriteOptimizedPreset());
+
+ await using var db = await Database.OpenAsync("ingest.db", options);
+
+ await db.ExecuteAsync("""
+ CREATE TABLE IF NOT EXISTS bench (
+ id INTEGER PRIMARY KEY,
+ value INTEGER,
+ text_col TEXT,
+ category TEXT
+ )
+ """);
+
+ var batch = db.PrepareInsertBatch("bench", initialCapacity: 1000);
+
+ await db.BeginTransactionAsync();
+ try
+ {
+ for (int block = 0; block < 100; block++)
+ {
+ for (int i = 0; i < 1000; i++)
+ {
+ int id = (block * 1000) + i;
+ batch.AddRow(
+ DbValue.FromInteger(id),
+ DbValue.FromInteger(id),
+ DbValue.FromText("durable_batch"),
+ DbValue.FromText("Alpha"));
+ }
+
+ await batch.ExecuteAsync();
+ }
+
+ await db.CommitAsync();
+ }
+ catch
+ {
+ await db.RollbackAsync();
+ throw;
+ }
+ The reason this matters is straightforward: it keeps the workload on the engine + bulk path instead of paying repeated SQL parse and planner cost for every row.
+Batch Commits Aggressively Before You Reach for Exotic Knobs
+The first major durable-write lever is still rows per commit.
+The practical starting guidance from the current benchmark matrix is:
+-
+
- Start at
1000rows per commit.
+ - Measure
10000only for dedicated ingest jobs that can tolerate larger + commit latency and larger WAL bursts.
+
Single-row durable auto-commit inserts are valid for correctness tests and for + latency-sensitive write APIs, but they are not the right baseline for bulk + ingest.
+Use the Write-Heavy Storage Preset
+For durable file-backed ingest, start with
+ UseWriteOptimizedPreset().
That is the current recommended storage baseline for write-heavy engine + workloads and is documented in both:
+-
+
- CSharpDB.Engine README +
- CSharpDB.Storage README +
Prefer Monotonic Primary Keys
+Monotonic primary keys are materially cheaper than random primary keys.
+Why:
+-
+
- they keep the table B-tree growing on the right edge +
- they reduce split-heavy locality loss +
- they make the primary insert path more predictable +
The durable batching matrix in this repo continues to show that random-key + primary inserts are still a major throughput cliff relative to monotonic keys.
+Keep Secondary Indexes Off the Hot Ingest Path When You Can
+If you only remember one design rule besides batching, it should be this one:
+Every extra secondary index costs you per-row write work.
+The recent Plan 5 work materially improved duplicate-bucket maintenance and + composite-index locality, but the remaining slope on realistic insert shapes is + still mostly secondary-index maintenance rather than the PK-only bulk path.
+If your workflow allows it, the easy win is still:
+-
+
- load data first +
- build or refresh indexes after the ingest step +
Use Prepared Commands at the Provider Layer
+If you are not using the engine directly and instead go through CSharpDB.Data,
+ the equivalent rule is to reuse prepared commands.
That looks like ordinary ADO.NET:
+-
+
- create one
DbCommand
+ - add parameters once +
- call
Prepare()
+ - update parameter values per execution +
This is the right analogue to SQLite's prepare-once, bind-many workflow.
+Use Concurrent Insert Mode Only for the Right Shape
+CSharpDB does have a concurrent insert mode:
+ImplicitInsertExecutionMode.ConcurrentWriteTransactions
But it is not a magic global acceleration switch.
+The measured guidance is:
+-
+
- Keep the default serialized insert mode for hot right-edge insert loops. +
- Measure concurrent insert mode when several tasks insert into disjoint
+ explicit key ranges on one shared
Database.
+ - If each writer needs multiple statements to commit atomically, use the + explicit write-transaction APIs instead. +
That is the difference between using a feature because it exists and using it + because the workload shape actually matches it.
+Treat Durable Group Commit as a Measure-First Lever
+UseDurableGroupCommit(...) can help when several writers are already
+ overlapping and you are willing to trade some latency for better flush sharing.
But it should come after:
+-
+
- batching +
- prepared work reuse +
- key-shape cleanup +
- avoiding unnecessary indexes +
In other words, do not use group commit as a substitute for basic workload + hygiene.
+The Honest Public Conclusion
+If you are turning this into a blog post, the cleanest conclusion is:
+-
+
- CSharpDB is now ahead of the matched SQLite baseline on the primary durable + monotonic bulk row on this runner. +
- That result is real because the comparison was aligned at the schema, + batching, and durability levels. +
- The remaining hard performance work is no longer the PK-only monotonic bulk + path. +
- The bigger remaining costs are secondary-index maintenance, random-key + locality, and hot right-edge multi-writer contention. +
That is a much stronger statement than "CSharpDB beats SQLite" because it is + specific enough to survive scrutiny.
+Reproducing the Main Comparisons
+Matched durable bulk rows:
+dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --durable-sql-batching-scenario BatchSweep_InsertBatch_B1000_Baseline_PkOnly_Monotonic --repeat 3 --repro
+ dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --durable-sql-batching-scenario BatchSweep_InsertBatch_B10000_Baseline_PkOnly_Monotonic --repeat 3 --repro
+ dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --sqlite-compare --repeat 3 --repro
+ Concurrent engine versus SQLite C API:
+dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --concurrent-sqlite-capi-compare --repeat 3 --repro
+ Concurrent ADO.NET provider comparison:
+dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --concurrent-adonet-compare --repeat 3 --repro
+ EF Core comparison:
+dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --efcore-compare --repeat 3 --repro
+ dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --efcore-compare-hybrid-shared-connection --repeat 3 --repro
+ dotnet run -c Release --project .\tests\CSharpDB.Benchmarks\CSharpDB.Benchmarks.csproj -- --efcore-compare-auto-open-close --repeat 3 --repro
+ Related Source Material
+If you want the lower-level internal notes behind this write-up, start here:
+-
+
- Existing comparison guide +
- Plan 5: Raw Rows/Sec Vs SQLite +
- Benchmark suite README +
- CSharpDB.Engine README +
- CSharpDB.Storage README +
Fulfillment Hub: A Guided CSharpDB Sample
+Most database samples are too small to teach much. A todo list can show inserts and queries, but it does not show what happens when a real application needs forms, reports, imports, exports, operational procedures, search, and semi-structured event data.
+samples/fulfillment-hub is designed to be the opposite. It gives you a small warehouse and order-fulfillment system with one working database and enough moving parts to see how CSharpDB features fit together.
What the Sample Creates
+Running the sample creates a fresh fulfillment-hub-demo.db database. The seeder loads more than SQL schema; it also stores the metadata-driven parts that Admin can run later.
-
+
- Relational tables for customers, products, inventory, orders, purchase orders, shipments, returns, carriers, suppliers, and audit events. +
- Views such as
order_fulfillment_board,low_stock_watch,purchase_order_receiving_board, andshipment_manifest_report_source.
+ - Stored procedures for allocation, receiving, shipment creation, returns, and operational stats. +
- Admin forms for order work, purchase-order receiving, and return intake. +
- Admin reports for the open order queue, low-stock watch, and shipment manifest. +
- Stored pipeline packages for CSV import, JSON import, and low-stock export. +
- Typed collections for scanner sessions and webhook archives. +
- A full-text index over operational playbooks. +
Step 1: Run the Seeder
+From the repository root, run the sample project:
+dotnet run --project samples\fulfillment-hub\FulfillmentHubSample.csprojEach run deletes and recreates the demo database in the build output folder:
+samples\fulfillment-hub\bin\Debug\net10.0\fulfillment-hub-demo.dbThe console output should show counts for forms, reports, procedures, saved queries, stored pipelines, pipeline runs, collections, and the full-text index. It also prints a few query results so you know the database was seeded successfully.
+ +Step 2: Start Admin Against the Sample
+CSharpDB Admin reads its local database from ConnectionStrings:CSharpDB. The easiest way to point Admin at the sample is to set the connection string for this terminal session and then run the Admin project.
$sampleDb = (Resolve-Path .\samples\fulfillment-hub\bin\Debug\net10.0\fulfillment-hub-demo.db).Path
+$env:ConnectionStrings__CSharpDB = "Data Source=$sampleDb"
+$env:CSharpDB__Transport = "direct"
+
+dotnet run --project src\CSharpDB.Admin\CSharpDB.Admin.csprojThe development launch profile uses these URLs:
+-
+
https://localhost:61816
+ http://localhost:61817
+
If those ports are already busy, run without the launch profile and choose a local URL:
+dotnet run --no-launch-profile --project src\CSharpDB.Admin\CSharpDB.Admin.csproj -- --urls http://127.0.0.1:6190Step 3: Confirm You Opened the Right Database
+When Admin opens, check the title bar and Object Explorer. You should see the Fulfillment Hub database path and a populated object tree.
+The useful signal is not just that tables exist. You should also see:
+-
+
FormswithOrder Workbench,Purchase Order Receiving, andReturn Intake.
+ ReportswithLow Stock Watch,Open Order Queue, andShipment Manifest.
+ Viewswith operational read models such asorder_fulfillment_boardandlow_stock_watch.
+ ProcedureswithAllocateOrder,ReceivePurchaseOrder,CreateShipment,RecordReturn, andRefreshOperationalStats.
+
Step 4: Start With the Order Board
+Open a query tab in Admin and run the main operational queue:
+SELECT order_number, customer_name, warehouse_code, order_status, required_ship_date
+FROM order_fulfillment_board
+WHERE order_status IN ('released', 'allocated', 'picking')
+ORDER BY required_ship_date, priority_code DESC, order_number;This is the first lesson in the sample: CSharpDB features are not isolated demos. The same order workflow is visible through a view, a saved query, a form, a report, and stored procedures.
+ +Step 5: Use the Forms
+Open Forms in the Object Explorer and start with Order Workbench. This form is bound to the orders table and includes child data for order lines.
Then open Purchase Order Receiving. That form is tied to the purchase-order model and helps you inspect the same receiving data that the procedure updates.
Finally, open Return Intake. It shows how reverse logistics can live beside outbound fulfillment without becoming a separate system.
Step 6: Run Operational Procedures
+Procedures let you package multi-statement operational work. You can run them from SQL tabs with EXEC commands.
EXEC RefreshOperationalStats;
+
+EXEC ReceivePurchaseOrder purchaseOrderId=9001;
+
+EXEC AllocateOrder orderId=7005;
+
+EXEC CreateShipment orderId=7001, shipmentId=8101, shipmentNumber='SHP-8101', carrierId=2;After each procedure, rerun the related view. This is where the sample starts to feel like a small operations system: receiving changes inventory, allocation changes order state, and shipment creation produces report-ready data.
+ +Step 7: Open the Reports
+The sample ships with three Admin reports. They are intentionally tied to the same operational story:
+-
+
Open Order Queueshows the work waiting on the floor.
+ Low Stock Watchshows shortage pressure by warehouse and SKU.
+ Shipment Manifestshows shipment-ready output from a report-shaped view.
+
Run CreateShipment, then open the shipment report. That connects the procedural workflow to the reporting surface.
Step 8: Inspect Pipelines
+The seeder stores and runs three pipeline packages:
+-
+
supplier-receipts-importimports CSV receiving data.
+ marketplace-orders-importimports JSON marketplace orders.
+ low-stock-exportexports the low-stock view to CSV.
+
In Admin, open the pipeline area and inspect the stored packages and run history. Then check the generated export file:
+samples\fulfillment-hub\bin\Debug\net10.0\generated-output\low-stock-watch.csvStep 9: Look at Collections and Full-Text Search
+Fulfillment Hub also includes data that does not need to be forced into normal relational tables. The sample seeds two collections:
+-
+
scanner_sessionsfor handheld scanner state.
+ webhook_archivefor external event payloads.
+
It also creates a full-text index called fts_ops_playbooks over the ops_playbooks table. That gives the sample a place for operational runbooks next to the live data.
The sample project demonstrates the full-text query through the engine API:
+await using var db = await Database.OpenAsync("samples/fulfillment-hub/bin/Debug/net10.0/fulfillment-hub-demo.db");
+var hits = await db.SearchAsync("fts_ops_playbooks", "partial receipt");How to Learn From the Sample
+The best way to use Fulfillment Hub is to treat it like a live system, not a collection of files.
+-
+
- Run the seeder. +
- Open the generated database in Admin. +
- Run one query against a view. +
- Open the matching form or report. +
- Run a procedure that changes the workflow. +
- Reopen the view, form, and report to see what changed. +
That loop is the point of the sample. It shows how CSharpDB can hold the transactional model, the operational read models, the metadata-driven UI, the reporting layer, the integration pipelines, and supporting document-style data in one local database.
+ + +Blog
Tutorials, guides, and best practices for building with CSharpDB.
Fulfillment Hub: A Guided CSharpDB Sample
+Seed a full warehouse and order-fulfillment database, then explore its tables, forms, reports, procedures, pipelines, collections, and full-text search through CSharpDB Admin.
+Changelog
Release announcements and project updates. For planned work, see the roadmap.
v3.4.0 — Admin Runtime, Fulfillment Hub Sample & Docs Polish
+Focuses on making CSharpDB easier to explore locally: a richer Admin experience, a complete Fulfillment Hub sample database, a run-only forms web host, and clearer docs for launching and learning from the project:
+-
+
- Fulfillment Hub sample — New end-to-end sample that seeds a working database with tables, views, stored procedures, forms, reports, pipelines, collections, and full-text search data. The walkthrough shows how to generate the database and open it through CSharpDB Admin. +
- Admin query workflow polish — Query autocomplete now handles
LIMIT,UPDATE ... SET,WHEREcolumn suggestions, andINSERT INTO ... columns ... VALUESflow more reliably.
+ - Designer usability fixes — Form designer property fields now respect the current light/dark theme, and Query Designer views include resizable split bars so long SQL previews and result grids can be adjusted without losing context. +
- Forms-only web host — Added a run-only Forms Web project path for displaying saved forms from a CSharpDB database without exposing design mode. Static assets, shared styling, and the themed home screen now load correctly when started through the provided script. +
- Local Admin launcher guidance — New blog documentation explains how to create a small C# launcher executable that starts
CSharpDB.Admin.exe, waits for the ASP.NET Core endpoint, and opens the default browser.
+ - Website documentation polish — Added the Fulfillment Hub sample blog post, improved tutorial code color-coding across the static docs pages, refreshed release/PR notes, and kept the sitemap/blog index aligned with the new content. +
docs/getting-started.md. For the shorter curated page, see Getting Started.Getting Started with CSharpDB
+This tutorial walks you through using CSharpDB from opening a database to running queries and transactions.
+Prerequisites
+-
+
- .NET 10 SDK +
Setup
+Add a project reference to CSharpDB.Engine (which transitively pulls in all other projects):
dotnet add reference path/to/src/CSharpDB.Engine/CSharpDB.Engine.csproj
+ Then add the using directive:
+using CSharpDB.Engine;
+ +
1. Opening a Database
+Database.OpenAsync opens an existing database file or creates a new one if it doesn't exist:
await using var db = await Database.OpenAsync("myapp.db");
+ The await using pattern ensures the database is properly closed when you're done. On open, if a WAL (Write-Ahead Log) file exists from a previous crash, the database automatically recovers committed data.
+
2. Creating Tables
+await db.ExecuteAsync(@"
+ CREATE TABLE products (
+ id INTEGER PRIMARY KEY,
+ name TEXT NOT NULL,
+ price REAL,
+ category TEXT
+ )
+ ");
+ Supported column types:
+-
+
INTEGER— 64-bit signed integer (long)
+ REAL— 64-bit floating point (double)
+ TEXT— UTF-8 string
+ BLOB— raw byte array
+
Column constraints:
+-
+
PRIMARY KEY— marks the column as the primary key
+ IDENTITY/AUTOINCREMENT— mark anINTEGER PRIMARY KEYas an identity column (explicit ID inserts are still allowed)
+ NOT NULL— disallows NULL values
+
Use IF NOT EXISTS to avoid errors when the table already exists:
await db.ExecuteAsync("CREATE TABLE IF NOT EXISTS products (id INTEGER, name TEXT)");
+ +
3. Inserting Data
+Insert with all columns:
+await db.ExecuteAsync("INSERT INTO products VALUES (1, 'Widget', 9.99, 'Hardware')");
+ await db.ExecuteAsync("INSERT INTO products VALUES (2, 'Gadget', 29.99, 'Electronics')");
+ await db.ExecuteAsync("INSERT INTO products VALUES (3, 'Doohickey', 4.99, 'Hardware')");
+ Insert with named columns (unspecified columns get NULL):
+await db.ExecuteAsync("INSERT INTO products (id, name) VALUES (4, 'Thingamajig')");
+ The return value of ExecuteAsync for DML statements includes a RowsAffected count:
var result = await db.ExecuteAsync("INSERT INTO products VALUES (5, 'Gizmo', 14.99, 'Electronics')");
+ Console.WriteLine($"Inserted {result.RowsAffected} row(s)"); // Inserted 1 row(s)
+ +
4. Querying Data
+SELECT all rows
+await using var result = await db.ExecuteAsync("SELECT * FROM products");
+
+ await foreach (var row in result.GetRowsAsync())
+ {
+ long id = row[0].AsInteger;
+ string name = row[1].AsText;
+ Console.WriteLine($" {id}: {name}");
+ }
+ ++Important: Use
+await usingon SELECT results to properly dispose the underlying operator chain.
Materialize all rows at once
+If you want all rows in a list instead of streaming:
+await using var result = await db.ExecuteAsync("SELECT * FROM products");
+ var rows = await result.ToListAsync();
+
+ Console.WriteLine($"Got {rows.Count} products");
+ WHERE clause
+await using var result = await db.ExecuteAsync(
+ "SELECT * FROM products WHERE price > 10.0 AND category = 'Electronics'");
+ Supported operators in WHERE: =, <>, <, >, <=, >=, AND, OR, NOT, +, -, *, /, LIKE, IN, BETWEEN, IS NULL, IS NOT NULL
LIKE pattern matching
+// % matches any sequence, _ matches any single character
+ await using var result = await db.ExecuteAsync(
+ "SELECT * FROM products WHERE name LIKE 'Gad%'");
+ IN lists
+await using var result = await db.ExecuteAsync(
+ "SELECT * FROM products WHERE category IN ('Hardware', 'Electronics')");
+ BETWEEN ranges
+await using var result = await db.ExecuteAsync(
+ "SELECT * FROM products WHERE price BETWEEN 5.0 AND 20.0");
+ ORDER BY
+await using var result = await db.ExecuteAsync(
+ "SELECT * FROM products ORDER BY price DESC");
+ You can sort by multiple columns and mix ASC/DESC:
await using var result = await db.ExecuteAsync(
+ "SELECT * FROM products ORDER BY category ASC, price DESC");
+ LIMIT and OFFSET
+await using var result = await db.ExecuteAsync(
+ "SELECT * FROM products ORDER BY price DESC LIMIT 3");
+
+ // Skip the first 5, then take the next 10
+ await using var paged = await db.ExecuteAsync(
+ "SELECT * FROM products ORDER BY id LIMIT 10 OFFSET 5");
+ Select specific columns
+await using var result = await db.ExecuteAsync(
+ "SELECT name, price FROM products WHERE price > 5.0");
+ System catalog metadata (sys.*)
+ You can inspect tables, columns, indexes, foreign keys, views, triggers, and object inventory with SQL:
+await using var tables = await db.ExecuteAsync(
+ "SELECT table_name, column_count, primary_key_column FROM sys.tables ORDER BY table_name");
+
+ await using var columns = await db.ExecuteAsync(
+ "SELECT column_name, data_type, is_nullable FROM sys.columns " +
+ "WHERE table_name = 'products' ORDER BY ordinal_position");
+
+ await using var objects = await db.ExecuteAsync(
+ "SELECT object_name, object_type, parent_table_name FROM sys.objects ORDER BY object_type, object_name");
+ Catalog sources:
+-
+
sys.tables
+ sys.columns
+ sys.indexes
+ sys.foreign_keys
+ sys.views
+ sys.triggers
+ sys.objects
+ sys.saved_queries
+
Underscored aliases are also available (sys_tables, sys_columns, etc.).
+
5. Aggregate Functions
+// COUNT
+ await using var r1 = await db.ExecuteAsync("SELECT COUNT(*) FROM products");
+
+ // SUM, AVG, MIN, MAX
+ await using var r2 = await db.ExecuteAsync(
+ "SELECT category, COUNT(*), AVG(price), MIN(price), MAX(price) FROM products GROUP BY category");
+
+ // HAVING
+ await using var r3 = await db.ExecuteAsync(
+ "SELECT category, COUNT(*) as cnt FROM products GROUP BY category HAVING cnt > 1");
+ Supported aggregate functions: COUNT(*), COUNT(col), COUNT(DISTINCT col), SUM, AVG, MIN, MAX
+
6. JOINs
+await db.ExecuteAsync("CREATE TABLE orders (id INTEGER, product_id INTEGER, qty INTEGER)");
+ await db.ExecuteAsync("INSERT INTO orders VALUES (1, 1, 10)");
+ await db.ExecuteAsync("INSERT INTO orders VALUES (2, 2, 5)");
+
+ // INNER JOIN
+ await using var result = await db.ExecuteAsync(@"
+ SELECT p.name, o.qty
+ FROM products p
+ INNER JOIN orders o ON p.id = o.product_id");
+
+ // LEFT JOIN (all products, even those without orders)
+ await using var result2 = await db.ExecuteAsync(@"
+ SELECT p.name, o.qty
+ FROM products p
+ LEFT JOIN orders o ON p.id = o.product_id");
+ Supported join types: INNER JOIN, LEFT JOIN, RIGHT JOIN, CROSS JOIN
+
7. Updating Rows
+var result = await db.ExecuteAsync(
+ "UPDATE products SET price = 12.99 WHERE name = 'Widget'");
+
+ Console.WriteLine($"Updated {result.RowsAffected} row(s)");
+ You can update multiple columns at once:
+await db.ExecuteAsync(
+ "UPDATE products SET price = 19.99, category = 'Premium' WHERE id = 2");
+ +
8. Deleting Rows
+var result = await db.ExecuteAsync("DELETE FROM products WHERE category = 'Hardware'");
+ Console.WriteLine($"Deleted {result.RowsAffected} row(s)");
+ Delete all rows (no WHERE clause):
+await db.ExecuteAsync("DELETE FROM products");
+ +
9. Working with NULL
+Insert a NULL value explicitly:
+await db.ExecuteAsync("INSERT INTO products VALUES (10, 'Mystery', NULL, NULL)");
+ Check for NULL when reading:
+await using var result = await db.ExecuteAsync("SELECT * FROM products");
+ await foreach (var row in result.GetRowsAsync())
+ {
+ if (row[2].IsNull)
+ Console.WriteLine($"{row[1].AsText}: no price set");
+ else
+ Console.WriteLine($"{row[1].AsText}: ${row[2].AsReal}");
+ }
+ Filter with IS NULL / IS NOT NULL:
+await using var result = await db.ExecuteAsync(
+ "SELECT * FROM products WHERE price IS NOT NULL");
+ +
10. Transactions
+By default, each DML/DDL statement auto-commits. For multi-statement atomicity, use explicit transactions:
+await db.BeginTransactionAsync();
+ try
+ {
+ await db.ExecuteAsync("INSERT INTO products VALUES (20, 'Item A', 5.00, 'Batch')");
+ await db.ExecuteAsync("INSERT INTO products VALUES (21, 'Item B', 7.50, 'Batch')");
+ await db.ExecuteAsync("INSERT INTO products VALUES (22, 'Item C', 3.25, 'Batch')");
+
+ await db.CommitAsync(); // All three inserts are now durable
+ }
+ catch
+ {
+ await db.RollbackAsync(); // None of the inserts are persisted
+ throw;
+ }
+ Transactions also improve performance for bulk inserts since dirty pages are written to the WAL only once rather than per-statement.
+Rollback example
+await db.BeginTransactionAsync();
+ await db.ExecuteAsync("DELETE FROM products"); // All rows deleted in memory
+ await db.RollbackAsync(); // Changes discarded — all rows restored
+ +
11. Indexes
+Create indexes to speed up equality lookups:
+// Regular index
+ await db.ExecuteAsync("CREATE INDEX idx_category ON products (category)");
+
+ // Unique index (enforces uniqueness)
+ await db.ExecuteAsync("CREATE UNIQUE INDEX idx_name ON products (name)");
+ When a WHERE clause contains column = value on an indexed column, the query planner automatically uses the index instead of a full table scan.
+
12. ALTER TABLE
+// Add a column
+ await db.ExecuteAsync("ALTER TABLE products ADD COLUMN weight REAL");
+
+ // Drop a column
+ await db.ExecuteAsync("ALTER TABLE products DROP COLUMN weight");
+
+ // Drop a foreign key constraint by name
+ await db.ExecuteAsync("ALTER TABLE child_orders DROP CONSTRAINT fk_child_orders_parent_id_abcd1234");
+
+ // Rename a column
+ await db.ExecuteAsync("ALTER TABLE products RENAME COLUMN category TO department");
+
+ // Rename a table
+ await db.ExecuteAsync("ALTER TABLE products RENAME TO inventory");
+ Use sys.foreign_keys to discover generated foreign key names before dropping one:
await using var foreignKeys = await db.ExecuteAsync(
+ "SELECT constraint_name, table_name, column_name FROM sys.foreign_keys ORDER BY table_name, column_name");
+ Retrofitting foreign keys onto older databases
+If you already have tables on disk without REFERENCES metadata, opening the database on a newer engine does not add foreign keys automatically. Use the maintenance migration workflow when you want to validate and then persist FK metadata onto existing tables:
using CSharpDB.Client;
+ using CSharpDB.Client.Models;
+
+ await using var client = CSharpDbClient.Create(new CSharpDbClientOptions
+ {
+ DataSource = "myapp.db"
+ });
+
+ var spec =
+ new[]
+ {
+ new ForeignKeyMigrationConstraintSpec
+ {
+ TableName = "orders",
+ ColumnName = "customer_id",
+ ReferencedTableName = "customers",
+ ReferencedColumnName = "id",
+ OnDelete = ForeignKeyOnDeleteAction.Cascade,
+ },
+ };
+
+ var preview = await client.MigrateForeignKeysAsync(new ForeignKeyMigrationRequest
+ {
+ ValidateOnly = true,
+ ViolationSampleLimit = 100,
+ Constraints = spec,
+ });
+
+ if (!preview.Succeeded)
+ {
+ foreach (var violation in preview.Violations)
+ Console.WriteLine($"{violation.TableName}.{violation.ColumnName}: {violation.Reason}");
+ }
+ else
+ {
+ await client.MigrateForeignKeysAsync(new ForeignKeyMigrationRequest
+ {
+ BackupDestinationPath = "pre-fk.backup.db",
+ Constraints = spec,
+ });
+ }
+ Notes:
+-
+
ValidateOnly = truepreviews the migration without changing schema or data.
+ BackupDestinationPathis optional but recommended for apply mode.
+ - The same operation is available through HTTP, gRPC, the CLI, and the Admin Storage tab. +
Schema migration pattern (PRIMARY KEY / IDENTITY changes)
+ALTER TABLE does not currently support changing an existing column to/from PRIMARY KEY or IDENTITY.
+ Use a create-copy-swap migration:
BEGIN TRANSACTION;
+
+ -- 1) Create target schema
+ CREATE TABLE users_v2 (
+ id INTEGER PRIMARY KEY IDENTITY,
+ name TEXT NOT NULL,
+ email TEXT NOT NULL
+ );
+
+ -- 2) Copy data
+ -- For small tables, insert explicit values:
+ INSERT INTO users_v2 (id, name, email) VALUES (1, 'Alice', 'alice@acme.io');
+ INSERT INTO users_v2 (id, name, email) VALUES (2, 'Bob', 'bob@acme.io');
+
+ -- 3) Swap tables
+ DROP TABLE users;
+ ALTER TABLE users_v2 RENAME TO users;
+
+ COMMIT;
+ Notes:
+-
+
- Explicit identity values are allowed, so preserving old IDs is supported. +
- If you omit
idduring insert, CSharpDB auto-generates identity values.
+ - Bulk
INSERT INTO ... SELECT ...copy is not yet supported, so large-table copy should be done via application code (read rows, then insert into the new table).
+
+
13. Views
+Views are named, reusable queries:
+await db.ExecuteAsync(@"
+ CREATE VIEW expensive_products AS
+ SELECT name, price FROM products WHERE price > 20.0");
+
+ // Query the view like a table
+ await using var result = await db.ExecuteAsync("SELECT * FROM expensive_products");
+ +
14. Common Table Expressions (CTEs)
+await using var result = await db.ExecuteAsync(@"
+ WITH high_value AS (
+ SELECT * FROM products WHERE price > 10.0
+ )
+ SELECT name, price FROM high_value ORDER BY price DESC");
+ +
15. Triggers
+Triggers execute SQL automatically when data changes:
+await db.ExecuteAsync("CREATE TABLE audit_log (action TEXT, product_name TEXT)");
+
+ await db.ExecuteAsync(@"
+ CREATE TRIGGER log_insert
+ AFTER INSERT ON products
+ BEGIN
+ INSERT INTO audit_log VALUES ('INSERT', NEW.name);
+ END");
+
+ // Now inserting into products automatically logs to audit_log
+ await db.ExecuteAsync("INSERT INTO products VALUES (50, 'Auto-logged', 9.99, 'Test')");
+ Supported trigger types:
+-
+
BEFORE INSERT,AFTER INSERT
+ BEFORE UPDATE,AFTER UPDATE
+ BEFORE DELETE,AFTER DELETE
+
Use NEW.column in INSERT/UPDATE triggers and OLD.column in UPDATE/DELETE triggers.
+
16. Concurrent Readers
+Create reader sessions for snapshot-isolated reads that don't block writes:
+// Take a snapshot of the current database state
+ using var reader = db.CreateReaderSession();
+
+ // Writer can continue modifying data — reader won't see changes
+ await db.ExecuteAsync("INSERT INTO products VALUES (99, 'New item', 1.0, 'Test')");
+
+ // Reader sees the database as it was when the snapshot was taken
+ await using var result = await reader.ExecuteReadAsync("SELECT COUNT(*) FROM products");
+ Multiple reader sessions can be active simultaneously.
++
17. Dropping Tables
+await db.ExecuteAsync("DROP TABLE products");
+ Use IF EXISTS to avoid errors:
await db.ExecuteAsync("DROP TABLE IF EXISTS products");
+ +
18. Error Handling
+CSharpDB throws CSharpDbException with a typed ErrorCode:
using CSharpDB.Primitives;
+
+ try
+ {
+ await db.ExecuteAsync("SELECT * FROM nonexistent");
+ }
+ catch (CSharpDbException ex) when (ex.Code == ErrorCode.TableNotFound)
+ {
+ Console.WriteLine($"Table not found: {ex.Message}");
+ }
+ catch (CSharpDbException ex) when (ex.Code == ErrorCode.SyntaxError)
+ {
+ Console.WriteLine($"SQL syntax error: {ex.Message}");
+ }
+ Error codes:
+| Code | +Meaning | +
|---|---|
TableNotFound |
+ Referenced table doesn't exist | +
TableAlreadyExists |
+ CREATE TABLE on an existing table (without IF NOT EXISTS) | +
ColumnNotFound |
+ Referenced column doesn't exist in the table | +
DuplicateKey |
+ INSERT with a rowid that already exists | +
SyntaxError |
+ Invalid SQL syntax | +
TypeMismatch |
+ Value type doesn't match expected type | +
ConstraintViolation |
+ NOT NULL, UNIQUE, or other constraint violated | +
IoError |
+ File system read/write failure | +
CorruptDatabase |
+ Database file structure is invalid | +
WalError |
+ Error reading/writing the WAL file | +
Busy |
+ Could not acquire write lock (another writer is active) | +
+
19. Using the ADO.NET Provider
+For standard .NET data access patterns, use the CSharpDB.Data package:
using CSharpDB.Data;
+
+ await using var conn = new CSharpDbConnection("Data Source=myapp.db");
+ await conn.OpenAsync();
+
+ // Parameterized queries
+ using var cmd = conn.CreateCommand();
+ cmd.CommandText = "SELECT * FROM products WHERE price > @minPrice";
+ cmd.Parameters.AddWithValue("@minPrice", 10.0);
+
+ await using var reader = await cmd.ExecuteReaderAsync();
+ while (await reader.ReadAsync())
+ {
+ long id = reader.GetInt64(0);
+ string name = reader.GetString(1);
+ double price = reader.GetDouble(2);
+ Console.WriteLine($"{id}: {name} (${price})");
+ }
+ The current provider routes ordinary embedded and daemon-backed usage through
+ the same authoritative CSharpDB.Client layer used by the other host surfaces.
+ That means you keep one provider API shape while choosing either a local
+ embedded database or a remote daemon endpoint in the connection string.
ExecuteScalar
+cmd.CommandText = "SELECT COUNT(*) FROM products";
+ var count = await cmd.ExecuteScalarAsync();
+ ExecuteNonQuery
+cmd.CommandText = "INSERT INTO products VALUES (100, 'New', 5.99, 'Test')";
+ int rowsAffected = await cmd.ExecuteNonQueryAsync();
+ Transactions via ADO.NET
+await using var txn = await conn.BeginTransactionAsync();
+ try
+ {
+ using var cmd = conn.CreateCommand();
+ cmd.Transaction = (CSharpDbTransaction)txn;
+ cmd.CommandText = "INSERT INTO products VALUES (200, 'Txn Item', 1.0, 'Test')";
+ await cmd.ExecuteNonQueryAsync();
+ await txn.CommitAsync();
+ }
+ catch
+ {
+ await txn.RollbackAsync();
+ throw;
+ }
+ Daemon-Backed ADO.NET
+If you want the same ADO.NET surface against a long-lived CSharpDB.Daemon
+ process, change only the connection string:
using CSharpDB.Data;
+
+ await using var conn = new CSharpDbConnection(
+ "Transport=Grpc;Endpoint=http://localhost:5820");
+ await conn.OpenAsync();
+
+ using var cmd = conn.CreateCommand();
+ cmd.CommandText = "SELECT COUNT(*) FROM products";
+ var count = (long)(await cmd.ExecuteScalarAsync() ?? 0L);
+ Notes:
+-
+
- embedded/local mode uses
Data Source=...
+ - daemon-backed mode uses
Transport=Grpc;Endpoint=http://...
+ - named shared in-memory connections such as
Data Source=:memory:sharedare process-local and do not cross the daemon boundary
+ NamedPipesis not implemented end to end yet, so useGrpcfor daemon access
+
+
20. Data Persistence
+Data survives application restarts — it's written to disk via the WAL on commit:
+// Session 1: Create and populate
+ await using (var db = await Database.OpenAsync("persistent.db"))
+ {
+ await db.ExecuteAsync("CREATE TABLE notes (id INTEGER, text TEXT)");
+ await db.ExecuteAsync("INSERT INTO notes VALUES (1, 'Remember this')");
+ }
+
+ // Session 2: Data is still there
+ await using (var db = await Database.OpenAsync("persistent.db"))
+ {
+ await using var result = await db.ExecuteAsync("SELECT * FROM notes");
+ var rows = await result.ToListAsync();
+ Console.WriteLine(rows[0][1].AsText); // "Remember this"
+ }
+ +
Next Steps
+-
+
- Architecture Guide — How the engine works layer by layer +
- Internals & Contributing — How to extend the engine, testing strategy +
- REST API Reference — Use CSharpDB over HTTP from any language +
- MCP Server Reference — Connect AI assistants to your database +
- CLI Reference — Interactive REPL with meta-commands +
- FAQ — Common setup and troubleshooting answers +
- Roadmap — Planned features and project direction +
Getting Started
Get CSharpDB up and running in your .NET project in minutes. Choose the API that fits your needs.
+Installation
Project Structure
├── docs/ │ ├── tutorials/native-ffi/ FFI tutorials (JavaScript via koffi, Python via ctypes) │ ├── tutorials/storage/ Storage tutorial track, study examples, and advanced standalone examples -│ ├── roadmap.md Product roadmap and status -│ └── rest-api.md REST host reference +│ └── roadmap.md Product roadmap and status │ └── samples/ Sample datasets + import helpers ├── ecommerce-store/ diff --git a/www/docs/performance-reference.html b/www/docs/performance-reference.html new file mode 100644 index 00000000..be35a17c --- /dev/null +++ b/www/docs/performance-reference.html @@ -0,0 +1,662 @@ + + + + + + + +docs/performance.md. For the shorter curated page, see Performance Guide.CSharpDB Performance Guide
+This guide is about getting real performance out of CSharpDB, not just turning knobs. + It is organized by workload scenario and grounded in the current benchmark and documentation set in this repo.
+Benchmark Basis
+This guide uses the latest current sources:
+-
+
- The latest perf guardrail report on March 29, 2026 passed
184/184checks on the benchmark runner.
+ - The broader macro, direct-client, and hybrid comparison captures were refreshed on March 28, 2026. +
- The published micro spot checks in
tests/CSharpDB.Benchmarks/README.mdandBenchmarkDotNet.Artifacts/results/provide the query-shape details used below.
+ - The benchmark README still includes some earlier March 29 narrative from a failing midday rerun; this guide uses the latest passing guardrail report as the current benchmark status. +
Benchmark environment for the published numbers:
+-
+
- CPU: Intel Core i9-11900K +
- OS: Windows 11 (
10.0.26300benchmark runner)
+ - Runtime: .NET 10 +
- Disk: NVMe SSD +
Treat the numbers here as directional for decision-making, then rerun the benchmark suite on your hardware before locking in a tuning choice.
+Start Here
+Use this table first. Most CSharpDB performance wins come from choosing the right mode and access path before touching advanced settings.
+| Scenario | +Start with | +Why | +
|---|---|---|
| Durable local app, mostly point reads | +File-backed + UseDirectLookupOptimizedPreset() |
+ Best simple baseline for hot local reads | +
| Durable app with a known hot working set | +Hybrid incremental-durable + HotTableNames / HotCollectionNames |
+ Best when you can pay open cost once and then hammer the same hot objects | +
| Ephemeral cache or periodic snapshot workflow | +In-memory + SaveToFileAsync when needed |
+ Biggest write throughput win by far | +
| Burst of related SQL reads | +Reuse one ReaderSession per reader burst |
+ Snapshot reuse is much cheaper than per-query session creation | +
| Ordered/range SQL queries | +Build indexes for the filter/sort columns and project only what you need | +Covered and compact paths are major wins | +
| Join/reporting workload | +Add join indexes and run ANALYZE after bulk changes |
+ Planner can use histograms, heavy hitters, composite-prefix stats, build-side choice, selective lookups, and bounded small-chain reordering | +
| Write-heavy ingest | +UseWriteOptimizedPreset() plus explicit transaction batching |
+ Batching is the biggest durable-write lever in the current suite | +
| Document/path queries | +Collection<T> + EnsureIndexAsync(...) on fields and paths you query |
+ Indexed collection paths are fast; unindexed predicate scans are not | +
| ADO.NET app with frequent opens | +Enable pooling explicitly | +Pooling dominates open/close overhead in the current provider benchmarks | +
The Biggest Levers
+If you remember only a few rules, use these:
+-
+
- Choose the right storage mode first. +
- Batch durable writes instead of auto-committing row-by-row. +
- Reuse
ReaderSessionfor bursts of related SQL reads.
+ - Create indexes that match the filter, join, and
ORDER BYshape.
+ - Prefer covered or narrow projections over
SELECT *.
+ - Run
ANALYZEafter bulk loads or major data-distribution changes.
+ - Measure before enabling advanced knobs like caching indexes, durable batch windows, or WAL preallocation. +
Scenario 1: Hot Point Reads In A Local Embedded App
+This is the default "application database" scenario: the app is already open, the database is local, and the hot path is GetAsync, primary-key lookups, or small indexed reads.
What to do
+-
+
- Start file-backed with
UseDirectLookupOptimizedPreset().
+ - Use
Collection<T>.GetAsync(...)for pure key/document access.
+ - Use
SELECT ... WHERE id = ...or equivalent indexed equality lookups for relational paths.
+ - Reuse a
ReaderSessionwhen you are doing many related reads from the same snapshot.
+
Why
+Current published steady-state numbers are already strong without exotic tuning:
+-
+
- File-backed SQL point lookups in the hot steady-state harness were about
1.10Mops/sec.
+ - File-backed collection point gets were about
1.78Mops/sec.
+ - Hot-cache micro references in the benchmark README still reach roughly
4.16MSQL PK lookups/sec and2.63Mcollection gets/sec on the benchmark runner.
+
For direct file-backed opens, the benchmark README's current recommendation is:
+-
+
builder.UseDirectLookupOptimizedPreset()for hot local lookup workloads
+
using CSharpDB.Engine;
+
+ var options = new DatabaseOptions()
+ .ConfigureStorageEngine(builder => builder.UseDirectLookupOptimizedPreset());
+
+ await using var db = await Database.OpenAsync("app.db", options);
+
+ await using var result = await db.ExecuteAsync(
+ "SELECT id, name FROM users WHERE id = 42");
+ When to use Collection<T>
+ Use the collection API when the workload is fundamentally:
+-
+
- key by id +
- document get/put/delete +
- indexed field equality +
- indexed path equality/range +
That path skips the SQL front door entirely.
+Scenario 2: Burst SQL Reads While Writes Continue
+This is the common local service pattern: a writer is active, but readers want a stable snapshot and tend to issue several related queries together.
+What to do
+-
+
- Create one
ReaderSessionper concurrent reader.
+ - Reuse that session for a burst of related reads. +
- Dispose each
QueryResultbefore issuing the next query on the same session.
+
using var reader = db.CreateReaderSession();
+
+ await using (var users = await reader.ExecuteReadAsync(
+ "SELECT id, name FROM users WHERE tier = 'gold'"))
+ {
+ while (await users.MoveNextAsync())
+ {
+ }
+ }
+
+ await using (var orders = await reader.ExecuteReadAsync(
+ "SELECT id, total FROM orders WHERE customer_id = 42"))
+ {
+ while (await orders.MoveNextAsync())
+ {
+ }
+ }
+ Why
+The current dedicated ReaderSessionBenchmarks show the setup penalty clearly:
-
+
- per-query reader-session point lookups are roughly
3xslower than reused-session lookups
+ - reused reader-session lookups are effectively at the same floor as direct
ExecuteAsync
+
The practical reading is simple:
+-
+
ReaderSessionis the right tool for concurrent snapshot reads.
+ - Reusing it is almost always better than recreating it for every small query. +
Scenario 3: Ordered And Range SQL Queries
+This is where query shape matters a lot. The main question is whether the engine can stay on index data, or whether it has to keep fetching wide base rows.
+What to do
+-
+
- Index the filter or sort column. +
- If possible, project only the indexed columns plus the row id. +
- Prefer narrow projections over
SELECT *.
+ - For multi-column filters, use composite indexes that match the left-to-right predicate shape. +
CREATE INDEX idx_orders_created_at ON orders(created_at);
+
+ SELECT id, created_at
+ FROM orders
+ WHERE created_at BETWEEN @from AND @to
+ ORDER BY created_at
+ LIMIT 100;
+ Why
+The current OrderByIndexBenchmarks and README spot checks show large wins when the query stays on index data:
-
+
ORDER BY value LIMIT 100on100Krows: +-
+
- index-order scan: about
64.79 us
+ - covered index-order scan: about
23.17 us
+
+ - index-order scan: about
WHERE value BETWEEN ...on100Krows: +-
+
- row fetch path: about
59.48 ms
+ - covered projection: about
18.49 ms
+
+ - row fetch path: about
- Composite covered projection on
100Krows: +-
+
- about
2.523 us
+
+ - about
This is one of the clearest themes in the repo's benchmarks:
+-
+
- the engine is fast at finding qualifying rows +
- the expensive part is often row fetch and materialization +
- covered and compact projections are a major lever +
Practical rule
+If the business endpoint only needs id, status, and created_at, do not ask CSharpDB to materialize twelve more columns.
Scenario 4: Reporting, Joins, And Selective Filters
+This is the "real SQL" scenario: non-trivial joins, selective predicates, and planner choice starts to matter.
+What to do
+-
+
- Create indexes on join keys and on selective filter columns. +
- Run
ANALYZEafter bulk loads and after major distribution changes.
+ - Inspect
sys.table_statsandsys.column_statswhen a plan is not behaving as expected.
+
CREATE INDEX idx_orders_customer ON orders(customer_id);
+ CREATE INDEX idx_order_items_order ON order_items(order_id);
+
+ ANALYZE;
+
+ SELECT * FROM sys.table_stats ORDER BY table_name;
+ SELECT * FROM sys.column_stats ORDER BY table_name, ordinal_position;
+ Why
+Current planner work already uses persisted stats for:
+-
+
- selective lookup choice +
- join method choice +
- hash build-side choice +
- limited inner-join reordering +
The benchmarks show this matters:
+-
+
INNER JOIN 1Kx20K (planner swap build side): about7.16 ms
+ INNER JOIN 1Kx20K (no swap via view): about11.43 ms
+ INNER JOIN on right PK (index nested-loop): about482.70 us
+ INNER JOIN on right PK (forced hash): about833.54 us
+
The practical rule:
+-
+
- if a reporting or join workload matters, index it and analyze it +
- if you skip
ANALYZE, you are leaving planner quality on the table
+
Scenario 5: Durable Write-Heavy Ingest
+This is where the wrong default usage hurts most. If you write one durable row at a time, the fixed WAL/fsync cost dominates.
+What to do
+-
+
- Start with
UseWriteOptimizedPreset().
+ - Batch writes in explicit transactions. +
- Measure with your real batch size before touching advanced durable-write knobs. +
using CSharpDB.Engine;
+
+ var options = new DatabaseOptions()
+ .ConfigureStorageEngine(builder => builder.UseWriteOptimizedPreset());
+
+ await using var db = await Database.OpenAsync("ingest.db", options);
+
+ await db.BeginTransactionAsync();
+ try
+ {
+ foreach (var row in rows)
+ {
+ await db.ExecuteAsync(
+ $"INSERT INTO events VALUES ({row.Id}, {row.TimestampTicks})");
+ }
+
+ await db.CommitAsync();
+ }
+ catch
+ {
+ await db.RollbackAsync();
+ throw;
+ }
+ Why
+The current durable batching benchmarks show that batching is the largest lever in the repo:
+-
+
- auto-commit single-row SQL: about
270.5ops/sec
+ - explicit transaction,
10rows/commit: about2,695.8rows/sec
+ - explicit transaction,
100rows/commit: about26,999.5rows/sec
+ - explicit transaction,
1000rows/commit: about197,256.9rows/sec
+
That is the core write-performance message for CSharpDB:
+-
+
- if you can batch, batch +
- if you cannot batch, expect durable single-row throughput to be much lower +
Preset guidance
+Current repo guidance is:
+-
+
- start with
UseWriteOptimizedPreset()
+ - do not assume
UseLowLatencyDurableWritePreset()is faster on your workload
+
On the current benchmark runner:
+-
+
- analyzed
UseWriteOptimizedPreset(): about267.8ops/sec
+ - analyzed
UseLowLatencyDurableWritePreset(): about261.4ops/sec
+
So the "low latency" preset is measure-first, not the default recommendation.
+Scenario 6: Multi-Writer Durable Contention
+This is narrower than normal ingest: several writer tasks are hitting the same shared Database instance.
What to do
+-
+
- Keep the baseline at
UseWriteOptimizedPreset().
+ - For
8in-process writers, benchmark WAL preallocation first.
+ - For
4in-process writers, benchmark a small durable batch window first.
+ - Do not cargo-cult batch-window tuning from the shared-writer case into the single-writer case. +
Why
+The current concurrent write diagnostics say:
+-
+
- best
8-writer row:W8_Batch0_Prealloc1MiBat about1078.6commits/sec
+ - close followers:
W8_Batch250usat1070.4,W8_Batch0at1068.2
+ - best
4-writer row:W4_Batch250usat about553.4commits/sec
+
But the single-writer durable diagnostics say the opposite for batch windows:
+-
+
BatchWindow(250us): about267.2ops/sec
+ BatchWindow(1ms): about65.8ops/sec
+
So:
+-
+
- shared-writer contention can justify small wait windows or preallocation +
- single-writer durable paths usually cannot +
Scenario 7: Cold File Reads, Tooling, And Cache-Pressured Workloads
+This is the "open a big local file and probe around" scenario: admin tools, inspection tools, one-shot analytics, or workloads that do not stay hot.
+What to do
+-
+
- Start with
UseDirectColdFileLookupPreset()for direct file-backed opens.
+ - If your hot pages are still living in the WAL, benchmark a small WAL read cache. +
- For deliberately bounded-cache workloads, start around
UseHybridFileCachePreset(2048)scale rather than tiny caches.
+
var options = new DatabaseOptions()
+ .ConfigureStorageEngine(builder => builder.UseDirectColdFileLookupPreset());
+
+ await using var db = await Database.OpenAsync("cold.db", options);
+ Why
+The current cold-read benchmarks show:
+-
+
- SQL cold lookup, copy-based path:
28.704 us
+ - SQL cold lookup, WAL-backed with
128-page WAL cache:19.83 us
+
The current file-backed collection tuning matrix also shows cache size matters:
+-
+
- indexed collection lookup at
16pages:54.15 us
+ - indexed collection lookup at
2048pages:20.57 us
+
Two practical takeaways:
+-
+
- cold local reads benefit from the mapped-file read path +
- tiny bounded caches are expensive unless the workload is truly memory-constrained +
What not to do
+Do not assume caching indexes are automatically a win.
+The current tuning matrix found UseCachingIndexes neutral-to-negative on these lookup workloads, including a case where a reused reader-session SQL lookup worsened from 41.36 us to 212.94 us.
Scenario 8: Known Hot Set With Durable Backing
+This is the hybrid-mode sweet spot: the database must stay durable on disk, but the same tables or collections are hot enough that prewarming them once is worth it.
+What to do
+-
+
- Use
OpenHybridAsync(...)withIncrementalDurable.
+ - Set
HotTableNamesand/orHotCollectionNamesfor the truly hot objects.
+ - Use this only when the process is long-lived enough to amortize open cost. +
await using var db = await Database.OpenHybridAsync(
+ "app.db",
+ new DatabaseOptions(),
+ new HybridDatabaseOptions
+ {
+ PersistenceMode = HybridPersistenceMode.IncrementalDurable,
+ HotTableNames = ["users", "orders"],
+ HotCollectionNames = ["sessions"]
+ });
+ Why
+In the dedicated resident hot-set harness:
+-
+
- hybrid hot-set incremental-durable SQL burst:
625.76Kops/sec
+ - hybrid hot-set incremental-durable collection burst:
707.38Kops/sec
+
But the trade-off is open cost:
+-
+
- hybrid hot-set SQL open only: about
87.39 ms
+ - hybrid hot-set collection open only: about
128.59 ms
+
This mode is excellent when:
+-
+
- open happens once +
- the process stays up +
- the same objects stay hot +
It is a bad fit when:
+-
+
- open latency is user-visible and frequent +
- the hot set is not stable +
Constraints
+Current hot-set warming is intentionally narrow:
+-
+
- supported only for
IncrementalDurable
+ - rejected for snapshot mode +
- rejected for bounded caches and custom page-cache factories +
Scenario 9: In-Memory Database As A Performance Tool
+This is the right answer when durability is optional during the hot path and you can snapshot explicitly.
+What to do
+-
+
- Use
OpenInMemoryAsync()for new ephemeral stores.
+ - Use
LoadIntoMemoryAsync()when you want to import a file, work in memory, then save back later.
+ - Use
SaveToFileAsync()at explicit persistence boundaries.
+
await using var db = await Database.OpenInMemoryAsync();
+
+ // hot path
+ await db.ExecuteAsync("INSERT INTO cache VALUES (1, 'hot')");
+
+ // persistence boundary
+ await db.SaveToFileAsync("cache.db");
+ Why
+In-memory mode is the strongest write-throughput lever in the repo:
+-
+
- hot steady-state in-memory SQL single insert: roughly
315Kops/sec
+ - hot steady-state in-memory collection single put: roughly
294Kops/sec
+ - in-memory batched SQL rows/sec is far above file-backed durable mode in the current macro suite +
Use it when the workflow is:
+-
+
- cache +
- local-first scratch workspace +
- import, process, export +
- periodically checkpointed embedded store +
Do not use it when every commit must already be durable before control returns.
+Scenario 10: Collection And Document Workloads
+The collection API can be very fast, but only if you use the indexed surfaces when the workload needs them.
+What to do
+-
+
- Use
GetAsyncfor key lookups.
+ - Use
EnsureIndexAsync(...)for fields or paths that you query repeatedly.
+ - Use
FindByIndexAsync(...),FindByPathAsync(...), andFindByPathRangeAsync(...)for indexed access.
+ - Keep the number of indexes honest; every extra index adds write maintenance cost. +
var users = await db.GetCollectionAsync<User>("users");
+
+ await users.EnsureIndexAsync(x => x.Email);
+ await users.EnsureIndexAsync("$.address.city");
+ await users.EnsureIndexAsync("$.tags[]");
+
+ await foreach (var match in users.FindByPathAsync("$.address.city", "Seattle"))
+ {
+ }
+ Why
+Current indexed collection path numbers are strong:
+-
+
FindByPath("$.address.city"): about568.9 ns
+ FindByPath("$.tags[]"): about474.8 ns
+ FindByPath("$.orders[].sku"): about599.3 ns
+ - integer path range with
1024matches: about552.79 us
+ - text path range with
1000matches: about545.69 us
+
But writes do pay for indexes:
+-
+
PutAsyncwith secondary indexes, insert case: about12.453 us
+ PutAsyncwith secondary indexes, update case: about29.079 us
+ DeleteAsyncwith secondary indexes: about47.314 us
+
Practical rule
+FindAsync(predicate) is convenient, but it is still a scan.
+ If the predicate matters to latency, promote it to an indexed field or indexed path.
Scenario 11: ADO.NET-Heavy Apps
+This is the integration scenario: existing .NET data-access code, ORMs, utilities, or apps that open and close connections frequently.
+What to do
+-
+
- Turn pooling on explicitly if you open and close connections often. +
- Use private
:memory:when you do not need cross-connection sharing.
+ - Use named shared
:memory:nameonly when you actually need multiple live connections against the same in-process memory database.
+ - Reuse command objects when it keeps your application code cleaner, but do not expect command preparation alone to be your biggest win. +
Data Source=myapp.db;Pooling=true;Max Pool Size=16
+ Data Source=:memory:
+ Data Source=:memory:shared-cache
+ Why
+The current ADO.NET benchmark results are clear:
+-
+
- open+close with pooling off: about
8.9 ms
+ - open+close with pooling on: about
1.9 us
+
That is the main provider-level lever.
+For in-memory ADO.NET:
+-
+
ExecuteScalaron private:memory::238.7 ns
+ ExecuteScalaron named shared:memory:name:347.1 ns
+ - insert on private
:memory::2.722 us
+ - insert on named shared
:memory:name:2.903 us
+
Prepared commands were roughly neutral in the current provider microbenchmarks:
+-
+
- parameterized select: about
298.1 us
+ - prepared select with reused command: about
296.7 us
+
So for most ADO.NET apps:
+-
+
- pooling matters a lot +
- memory mode choice matters a little +
- command preparation is not the first lever to chase +
What Usually Hurts
+These are the common ways to leave performance on the floor in CSharpDB:
+-
+
- Auto-committing every durable row when batching is possible. +
- Recreating a
ReaderSessionfor every small read.
+ - Using
SELECT *on range and ordered queries that could be covered or at least narrow.
+ - Skipping
ANALYZEafter bulk loads, then blaming the planner.
+ - Using
FindAsync(...)on large collections for fields that should be indexed.
+ - Turning on advanced knobs like caching indexes, durable batch windows, or WAL preallocation without measuring the exact scenario they are meant for. +
- Paying hybrid hot-set open cost for short-lived processes. +
- Using named shared
:memory:when private:memory:would do.
+
Recommended Workflow
+When tuning a real CSharpDB workload, use this order:
+-
+
- Pick the storage mode: file-backed, hybrid, or in-memory. +
- Pick the API surface: SQL, collection API, or ADO.NET. +
- Add the right indexes. +
- Shape queries so they stay narrow and covered when possible. +
- Run
ANALYZE.
+ - Reuse
ReaderSessionor pooled connections where appropriate.
+ - Batch writes. +
- Only then benchmark advanced knobs. +
Source Map
+Primary references for this guide:
+-
+
- Benchmark Suite +
- Latest Guardrail Report +
- Engine README +
- ADO.NET Provider README +
- Architecture Guide +
- Roadmap +
On This Page
Performance Guide
Getting real performance out of CSharpDB, not just turning knobs. Organized by workload scenario with functional code examples grounded in the current benchmark suite.
+ETL Pipelines
Built-in pipeline runtime for Extract-Transform-Load workflows. Define packages in JSON, validate them, serialize them, execute them in batches, and inspect run metrics.
+Overview
The pipeline system connects sources to destinations through a chain of transforms. Pipelines are defined as JSON package files and executed by the PipelineOrchestrator. The built-in runtime supports CSV and JSON file connectors.
docs/query-execution-pipeline.md. For the shorter curated page, see ETL Pipelines.Query Execution Pipeline
+This document describes how CSharpDB processes a SQL query from text to results. It is + intended for contributors and advanced users who want to understand how the engine makes + planning decisions.
++
Pipeline Overview
+SQL Text
+ │
+ ▼
+ ┌──────────────┐
+ │ Tokenizer │ CSharpDB.Sql/Tokenizer.cs
+ └──────┬───────┘
+ │ Token stream
+ ▼
+ ┌──────────────┐
+ │ Parser │ CSharpDB.Sql/Parser.cs
+ └──────┬───────┘
+ │ Abstract Syntax Tree (AST)
+ ▼
+ ┌──────────────┐
+ │Query Planner │ CSharpDB.Execution/QueryPlanner.cs
+ │ │ CSharpDB.Execution/CardinalityEstimator.cs
+ └──────┬───────┘
+ │ Physical operator tree
+ ▼
+ ┌──────────────┐
+ │ Execution │ CSharpDB.Execution/Operators.cs
+ └──────┬───────┘
+ │ Row-at-a-time or batched iteration
+ ▼
+ Query Result
+ +
1. Tokenization
+The tokenizer (Tokenizer.cs) performs a single pass over the SQL string, producing a
+ stream of Token(TokenType, string value, int position) objects.
-
+
- 120+ token types covering keywords, literals, operators, identifiers, and punctuation +
- Case-insensitive keyword matching via dictionary lookup +
- Handles SQL comments (
-- ...), string escaping (''), and parameter tokens (@name)
+ - Classifies numeric literals as
IntegerLiteralorRealLiteral
+
+
2. Parsing
+The parser (Parser.cs) is a recursive-descent parser that consumes tokens and builds a
+ typed AST.
AST Node Categories
+Statements — top-level SQL commands:
+-
+
- DDL:
CreateTableStatement,AlterTableStatement,DropTableStatement, +CreateIndexStatement,DropIndexStatement,CreateViewStatement,DropViewStatement, +CreateTriggerStatement,DropTriggerStatement,AnalyzeStatement
+ - DML:
InsertStatement,UpdateStatement,DeleteStatement
+ - Query:
SelectStatement,CompoundSelectStatement(set operations),WithStatement(CTEs)
+
Table references:
+-
+
SimpleTableRef(TableName, Alias)— single table or view
+ JoinTableRef(Left, Right, JoinType, Condition)— binary join tree
+
Expressions:
+-
+
LiteralExpression,ParameterExpression,ColumnRefExpression
+ BinaryExpression— arithmetic, comparison, logical operators
+ UnaryExpression—NOT, unary minus
+ FunctionCallExpression— aggregates and scalar functions
+ LikeExpression,InExpression,BetweenExpression,IsNullExpression
+ ScalarSubqueryExpression,InSubqueryExpression,ExistsExpression
+ CollateExpression
+
Parser Fast Paths
+The parser detects common query shapes and produces lightweight metadata instead of a full + AST, skipping unnecessary allocation:
+| Fast Path | +Pattern Detected | +Benefit | +
|---|---|---|
| Simple SELECT | +Single table, simple WHERE | +Skips full AST construction | +
| Primary key lookup | +SELECT ... WHERE pk = value |
+ Direct lookup metadata | +
| Simple INSERT | +INSERT INTO ... VALUES (...) |
+ Lightweight insert path | +
+
3. Query Planning
+The QueryPlanner dispatches statements to type-specific handlers. For SELECT queries,
+ it classifies the query into a SelectPlanKind to choose an execution strategy:
| Plan Kind | +Description | +
|---|---|
FastPrimaryKeyLookup |
+ Direct integer PK lookup — fastest path | +
FastIndexedLookup |
+ Equality lookup via index | +
FastSimpleTableScan |
+ Full scan with basic filtering | +
SimpleCountStar |
+ COUNT(*) using cached row count (no scan) |
+
SimpleScalarAggregateColumn |
+ Single aggregate on a column | +
SimpleGroupedIndexAggregate |
+ GROUP BY on indexed column (streaming) | +
General |
+ Full operator tree for complex queries | +
Plan classifications are cached (up to 1024 entries) so repeated queries skip re-analysis.
+Cardinality Estimation
+When ANALYZE has been run on a table, the CardinalityEstimator uses collected statistics
+ to estimate result sizes and guide operator selection.
Statistics collected by ANALYZE:
+| Statistic | +Scope | +Description | +
|---|---|---|
| Distinct count | +Per column | +Number of unique values | +
| Non-NULL count | +Per column | +Number of non-NULL values | +
| Min / Max value | +Per column | +Value range | +
| Frequent values | +Per column | +Top 8 values by occurrence | +
| Histogram buckets | +Per column | +16 quantile-based buckets for range estimation | +
| Prefix distinct counts | +Per index | +Distinct values for each prefix of a composite index | +
Estimation methods:
+-
+
- Lookup selectivity:
tableRows / distinctCountfor uniform distribution, adjusted + for skew when the lookup value matches a known frequent value
+ - Filter selectivity: Per-column selectivity multiplied across AND-conjuncts. Supports + equality, range, NULL, and discrete value list predicates. +
- Join cardinality:
(leftRows × rightRows) / max(leftDistinct, rightDistinct), + adjusted by non-NULL fraction and outer join semantics.
+
Without statistics, the planner uses heuristic fallback estimates.
+Index Selection
+For each AND-separated predicate in the WHERE clause, the planner checks whether an + available index can satisfy it:
+| Candidate | +Rank | +Description | +
|---|---|---|
| Integer primary key equality | +0 | +Best — direct B-tree lookup | +
| Unique index equality | +1 | +Single-row guarantee | +
| Non-unique index equality | +2 | +Accepted only if estimated rows ≤ 25% of table | +
When multiple candidates match, the planner picks the lowest rank. Ties are broken by + estimated row count (fewer is better). Composite indexes are used when equality predicates + match the index prefix columns.
+Join Operator Selection
+The planner tries join operators in preference order:
+1. Index Nested-Loop Join
+Requirements: INNER or LEFT join, right side is a simple table, equi-join condition exists, + and the right table has an index on the join key.
+Cost decision (when statistics are available):
+-
+
- Unique index: use if
leftRows ≤ rightRows × 8(PK) orleftRows ≤ rightRows × 2+ (unique)
+ - Non-unique index: use if estimated lookup cost < hash join cost +
Produces IndexNestedLoopJoinOperator (integer PK) or HashedIndexNestedLoopJoinOperator
+ (text/composite keys).
2. Hash Join
+Requirements: equi-join condition with extractable key columns.
+Build side selection: for INNER joins, builds the smaller estimated side. For LEFT/RIGHT + joins, respects outer table preservation.
+3. Nested-Loop Join (fallback)
+Cartesian product with join condition filtering. Used when no index or equi-join condition + is available.
+Join Reordering
+For queries joining 4–6 tables, the planner applies dynamic-programming-based join + reordering to find the lowest-cardinality ordering. Results are cached per query shape. + Views, CTEs, and system tables are excluded from reordering.
++
4. Execution
+Iterator Model
+Operators implement a pull-based iterator protocol:
+interface IOperator : IAsyncDisposable
+ {
+ ColumnDefinition[] OutputSchema { get; }
+ ValueTask OpenAsync();
+ ValueTask<bool> MoveNextAsync();
+ DbValue[] Current { get; }
+ }
+ The root operator is opened, then MoveNextAsync is called repeatedly until it returns
+ false. Results flow from leaf operators (scans) upward through the tree.
Batch Execution
+Performance-critical operators also implement IBatchOperator, which yields RowBatch
+ objects containing up to 64 rows per batch. This enables SIMD-friendly memory layout and
+ reduces per-row overhead.
interface IBatchOperator : IAsyncDisposable
+ {
+ ValueTask OpenAsync();
+ ValueTask<bool> MoveNextBatchAsync();
+ RowBatch CurrentBatch { get; }
+ }
+ Expression Evaluation
+Expressions are evaluated through two paths:
+| Path | +Used When | +How | +
|---|---|---|
| Interpreter | +One-off evaluation | +Recursive AST walk with per-row schema lookups | +
| Compiled | +Hot paths (filters, projections) | +AST compiled to Func<DbValue[], DbValue> delegate with bound column indices. Cached (up to 4096 entries). |
+
Operator Catalog
+Scan Operators
+| Operator | +Description | +
|---|---|
TableScanOperator |
+ Sequential B+tree scan. Supports pre-decode filtering and projection pushdown. | +
IndexScanOperator |
+ Ordered index scan with optional range bounds. | +
IndexOrderedScanOperator |
+ Prefix scan with secondary range filtering. | +
PrimaryKeyLookupOperator |
+ Direct row lookup by integer PK. Fastest scan path. | +
UniqueIndexLookupOperator |
+ Single-row lookup via unique index. | +
Projection variants (PrimaryKeyProjectionLookupOperator,
+ IndexScanProjectionOperator, etc.) fuse column selection into the scan to skip
+ unnecessary deserialization.
Join Operators
+| Operator | +Description | +
|---|---|
HashJoinOperator |
+ Hash table on build side, probed by probe side. Supports INNER, LEFT, RIGHT, CROSS. | +
IndexNestedLoopJoinOperator |
+ Probes right table via integer PK index for each left row. | +
HashedIndexNestedLoopJoinOperator |
+ Probes via hash-based index (text/composite keys). | +
NestedLoopJoinOperator |
+ Cartesian product with condition filtering. | +
Filter and Projection
+| Operator | +Description | +
|---|---|
FilterOperator |
+ Applies WHERE predicates. Optional compiled filter for hot paths. | +
ProjectionOperator |
+ Column selection and expression evaluation. | +
FilterProjectionOperator |
+ Fused filter + projection in a single pass. | +
Aggregation
+| Operator | +Description | +
|---|---|
HashAggregateOperator |
+ Grouped aggregation via hash table (GROUP BY). | +
ScalarAggregateOperator |
+ Single-row aggregation (no GROUP BY). | +
IndexGroupedAggregateOperator |
+ Streaming GROUP BY on indexed column (no hash table). | +
CompositeIndexGroupedAggregateOperator |
+ Streaming GROUP BY on composite index prefix. | +
CountStarTableOperator |
+ Returns cached row count — no scan needed. | +
Additional specialized variants exist for specific aggregate patterns
+ (ScalarAggregateLookupOperator, FilteredScalarAggregateTableOperator, etc.).
Sort and Limit
+| Operator | +Description | +
|---|---|
SortOperator |
+ Full sort (in-memory for small sets, merge sort for larger). | +
TopNSortOperator |
+ Heap-based top-N for ORDER BY ... LIMIT N — avoids full sort. |
+
LimitOperator |
+ Truncates output after N rows. | +
OffsetOperator |
+ Skips first N rows. | +
Other
+| Operator | +Description | +
|---|---|
DistinctOperator |
+ Removes duplicates via hash set. | +
MaterializedOperator |
+ Wraps pre-materialized rows (CTEs, subquery results). | +
Optimization Interfaces
+Operators expose capability interfaces that enable cross-operator optimization:
+| Interface | +Purpose | +
|---|---|
IPreDecodeFilterSupport |
+ Push equality filters into the decode layer, eliminating rows before full deserialization | +
IProjectionPushdownTarget |
+ Declare needed output columns so scans skip unused columns | +
IEstimatedRowCountProvider |
+ Expose cardinality to parent operators for cost decisions | +
IRowBufferReuseController |
+ Control whether row buffers are reused or cloned | +
+
5. Worked Example
+SELECT c.name, SUM(o.amount) AS total
+ FROM customers c
+ LEFT JOIN orders o ON o.cust_id = c.id
+ WHERE c.country = 'USA'
+ GROUP BY c.id, c.name
+ ORDER BY total DESC
+ LIMIT 10;
+ Planning decisions:
+-
+
-
+
Index check on
+customers.country: Non-unique index found. Estimated 300K rows out + of 1M — selectivity passes the 25% threshold → useIndexScanOperator.
+ -
+
Join strategy: Index exists on
+orders.cust_id. Estimated left rows (300K) ≤ + right rows × 8 → useIndexNestedLoopJoinOperator.
+ -
+
Aggregation: GROUP BY columns not aligned with an index →
+HashAggregateOperator.
+ -
+
Sort + limit:
+ORDER BY total DESC LIMIT 10→TopNSortOperator(heap of size 10, + avoids sorting all groups).
+
Resulting operator tree:
+LimitOperator (10)
+ └─ TopNSortOperator (total DESC, N=10)
+ └─ HashAggregateOperator (GROUP BY c.id, c.name; SUM(o.amount))
+ └─ IndexNestedLoopJoinOperator (LEFT, o.cust_id = c.id)
+ ├─ IndexScanOperator (customers, country = 'USA')
+ └─ orders.cust_id index
+ Execution:
+-
+
IndexScanOperatoriterates thecountryindex for'USA'entries, yielding customer + rows in batches of 64.
+ - For each customer,
IndexNestedLoopJoinOperatorprobes the orders index oncust_id, + yielding joined rows (or a NULL-padded row for LEFT JOIN when no orders exist).
+ HashAggregateOperatoraccumulatesSUM(amount)per(id, name)group in a hash + table, then emits all groups.
+ TopNSortOperatormaintains a min-heap of 10 entries bytotal DESC, discarding rows + below the current 10th largest.
+ LimitOperatoryields the final 10 rows to the caller.
+
+
Constants
+| Constant | +Value | +Purpose | +
|---|---|---|
| Default batch size | +64 rows | +RowBatch capacity |
+
| Expression cache | +4096 entries | +Compiled expression delegate cache | +
| Plan cache | +1024 entries | +SELECT plan classification cache | +
| Histogram buckets | +16 | +Quantile buckets per column in ANALYZE | +
| Frequent values | +8 | +Top-N tracked per column in ANALYZE | +
| Max join reorder leaves | +6 | +DP-based reordering table limit | +
| Max trigger depth | +16 | +Recursive trigger invocation limit | +
| Max FK cascade depth | +64 | +Foreign key cascade recursion limit | +
On This Page
CSharpDB REST API Reference
-The CSharpDB REST API exposes the full database feature set over HTTP, enabling cross-language interoperability. Built with ASP.NET Core Minimal APIs, it includes OpenAPI documentation and an interactive Scalar UI.
+The CSharpDB REST API is now hosted by CSharpDB.Daemon by default. The daemon exposes the existing HTTP /api surface and the gRPC service from one long-running process backed by the same warm database client.
The REST surface enables cross-language interoperability over JSON/HTTP. Built with ASP.NET Core Minimal APIs, it includes OpenAPI documentation and an interactive Scalar UI when the daemon runs in Development mode.
Running the API
-dotnet run --project src/CSharpDB.ApiThe API starts on http://localhost:61818 (HTTP) and https://localhost:61819 (HTTPS).
Interactive documentation: Open http://localhost:61818/scalar/v1 in a browser to explore and test endpoints with the Scalar API explorer.
Run the combined daemon host from source:
+dotnet run --project src/CSharpDB.Daemon/CSharpDB.Daemon.csprojThe daemon launch profile starts on https://localhost:49995 and http://localhost:49996. REST endpoints are available under /api, and gRPC endpoints are available from the same host.
Interactive documentation: In Development mode, open https://localhost:49995/scalar/v1 or http://localhost:49996/scalar/v1 in a browser to explore and test endpoints with the Scalar API explorer.
For a stable local HTTP port, set ASPNETCORE_URLS explicitly before starting the daemon:
$env:ConnectionStrings__CSharpDB = "Data Source=C:\data\sample.db"
+$env:ASPNETCORE_URLS = "http://localhost:5820"
+dotnet run --project src/CSharpDB.Daemon/CSharpDB.Daemon.csprojWith that override, the REST base URL is http://localhost:5820/api and Scalar is available at http://localhost:5820/scalar/v1.
Configuration
-The default database path is configured in src/CSharpDB.Api/appsettings.json:
The current default database path and daemon host behavior are configured in src/CSharpDB.Daemon/appsettings.json:
{
"ConnectionStrings": {
"CSharpDB": "Data Source=csharpdb.db"
+ },
+ "CSharpDB": {
+ "Daemon": {
+ "EnableRestApi": true
+ },
+ "HostDatabase": {
+ "OpenMode": "HybridIncrementalDurable",
+ "ImplicitInsertExecutionMode": "ConcurrentWriteTransactions",
+ "UseWriteOptimizedPreset": true,
+ "HotTableNames": [],
+ "HotCollectionNames": []
+ }
}
}CSharpDB:Daemon:EnableRestApi controls whether the daemon maps the REST /api surface. The default is true. Set it to false only when the daemon should expose gRPC without REST.
CORS is enabled for all origins by default (development convenience). JSON responses use camelCase naming and omit null values.
+The standalone CSharpDB.Api project remains available for REST-only hosting, but the recommended remote host is CSharpDB.Daemon so REST and gRPC clients share the same warm database process.
Endpoints
All endpoints are prefixed with /api.
POST /api/maintenance/migrate-foreign-keys
validateOnly = truepreviews the migration without mutating schema or data.backupDestinationPathis optional and is only used during apply mode.
- - Paths are resolved on the API host machine, not on the caller. +
- Paths are resolved on the daemon host machine, not on the caller.
Tables
@@ -525,6 +546,7 @@Error Handling
See Also
- Getting Started Tutorial — Engine API walkthrough +
- Multi-Writer gRPC Daemon — Daemon runtime model and remote-host guidance
- Storage Architecture Deep Dive — How the engine works internally
- CLI Reference — Interactive REPL commands
- Sample Datasets — Ready-to-run SQL scripts diff --git a/www/docs/sql-reference.html b/www/docs/sql-reference.html new file mode 100644 index 00000000..54bd2753 --- /dev/null +++ b/www/docs/sql-reference.html @@ -0,0 +1,601 @@ + + + + + + + +
INSERTtriggers:NEWis available
+ DELETEtriggers:OLDis available
+ UPDATEtriggers: bothNEWandOLDare available
+ CASE/WHENexpressions
+ CASTexpressions (implicit coercion only)
+ DEFAULTcolumn values
+ CHECKconstraints
+ RETURNINGclause on INSERT/UPDATE/DELETE
+ UPSERT/ON CONFLICT/INSERT OR REPLACE
+ - Recursive CTE execution (
WITH RECURSIVEis parsed but not evaluated)
+ - String functions (
UPPER,LOWER,LENGTH,SUBSTR,TRIM)
+ - Date/time functions (
DATE,TIME,DATETIME,STRFTIME)
+ - Math functions (
ABS,ROUND,CEIL,FLOOR)
+ - Window functions (
OVER,PARTITION BY,ROW_NUMBER, etc.)
+ - Stored procedures +
- Composite primary keys / composite foreign keys +
- Architecture Overview +
- FileStorageDevice +
- IStorageDevice Interface +
- Device Scenarios
+
-
+
- 1. Create a New File +
- 2. Open an Existing File +
- 3. Write Raw Bytes at an Offset +
- 4. Read Raw Bytes from an Offset +
- 5. Read Past End of File (Zero-Fill Behavior) +
- 6. Pre-allocate / Extend File Length +
- 7. Flush to Disk (fsync) +
- 8. Check File Length +
- 9. Dispose Synchronously +
- 10. Dispose Asynchronously +
- 11. Cancellation Support +
- 12. Injecting via IStorageDevice (Testability) +
- 13. Writing Fixed-Size Pages (4 KB) +
- 14. Reading Fixed-Size Pages (4 KB) +
- 15. Appending Sequential Pages +
+ - Pager + + +
- B+Tree + + +
- Write-Ahead Log (WAL) + + +
- Slotted Page Layout + + +
- Indexing + + +
- Record Serialization + + +
- Schema Catalog + + +
- Folder & File Storage
+
-
+
- Domain Models +
- F1. Bootstrap the Storage +
- F2. Create a Folder +
- F3. Create a File Inside a Folder +
- F4. Read a File +
- F5. List All Files in a Folder +
- F6. List All Folders +
- F7. Update File Content +
- F8. Delete a File +
- F9. Delete a Folder and Its Contents +
- F10. Rename or Move a File +
- F11. Search Files by Predicate +
- F12. Bulk Create with an Explicit Transaction +
- F13. SQL-Based Approach +
- F14. One Database File per Folder (Multi-Volume) +
+ - Key Design Notes +
- Memory-mapped reads for clean main-file pages when the storage device supports it +
- Speculative B+tree leaf read-ahead during sequential forward scans +
- Checkpoint residency preservation so already-owned main-file pages stay hot across checkpoint in lazy-resident hybrid engine mode +
- True async I/O via
RandomAccess.ReadAsync/RandomAccess.WriteAsync
+ - Position-independent reads and writes -- no seek, no shared file pointer +
- Concurrent reads from other processes (
FileShare.Read)
+ - Direct fsync to durable storage via
RandomAccess.FlushToDisk
+ - Folders ->
"/documents/reports"
+ - Files ->
"/documents/reports/summary.txt"
+ - Architecture Guide +
- Storage Package README +
- Storage Tutorial Track +
- Advanced Storage Examples +
- Engine README +
- Storage Diagnostics README +
- Benchmark Suite +
- Roadmap +
- Single-file database with 4 KB page-oriented storage +
- B+tree-backed tables and secondary indexes +
- Write-Ahead Log with crash recovery and auto-checkpoint +
- Concurrent snapshot-isolated readers via WAL-based MVCC +
- Full SQL pipeline: tokenizer, parser, query planner, operator tree +
- JOINs (INNER, LEFT, RIGHT, CROSS), aggregates, GROUP BY, HAVING, CTEs +
- Set operations:
UNION,INTERSECT,EXCEPT
+ SELECT DISTINCTand DISTINCT aggregates
+ - Scalar subqueries,
IN (SELECT ...), andEXISTS (SELECT ...), including correlated evaluation in filters, non-aggregate projections, andUPDATE/DELETEexpressions
+ - Scalar
TEXT(expr)for filter-friendly text coercion
+ - Composite (multi-column) indexes +
- Ordered integer index range scans (
<,<=,>,>=,BETWEEN) in the fast lookup path
+ ANALYZE, persistedsys.table_stats/sys.column_stats, and stale-aware column-stat refresh
+ - Phase-2 cost-based query planning: statistics-guided access-path selection, join method choice, hash build-side choice, histogram/heavy-hitter/cardinality estimation, composite-prefix correlation modeling, and bounded small-chain inner-join reordering +
- SQL statement and SELECT plan caching +
- First-class
IDENTITY/AUTOINCREMENTsupport forINTEGER PRIMARY KEYcolumns
+ - Persisted table
NextRowIdhigh-water mark with compatibility fallback for legacy metadata
+ - Views and triggers (BEFORE/AFTER on INSERT/UPDATE/DELETE) +
- Foreign key constraints: single-column, column-level
REFERENCESwith optionalON DELETE CASCADE
+ - Older-database foreign-key retrofit migration across direct, HTTP, gRPC, CLI, and Admin +
- ADO.NET provider (DbConnection, DbCommand, DbDataReader, DbTransaction) +
- ADO.NET
GetSchema()metadata collections forMetaDataCollections,Tables,Columns,Indexes,Views, andForeignKeys
+ - ADO.NET connection pooling with
ClearPool/ClearAllPools
+ - In-memory database mode with explicit load-from-disk and save-to-disk APIs +
- Shared/private in-memory ADO.NET connections with named shared-memory hosts +
- Document Collection API (NoSQL) with typed Put/Get/Delete/Scan/Find +
- Collection UTF-8 payload fast path with compatibility for legacy backing rows +
- Collection secondary field indexes via
EnsureIndexAsync/FindByIndexAsync
+ - Maintenance report,
REINDEX, andVACUUMflows across client, CLI, API, and Admin UI
+ - Dedicated
CSharpDB.DaemongRPC host for remoteCSharpDB.Clientaccess
+ - Remote host consolidation in
CSharpDB.Daemon, with REST/apiand gRPC sharing the same warm hosted database client
+ - Storage tuning presets, bounded WAL read caching, memory-mapped main-file reads, and sliced background WAL auto-checkpointing +
- SQL executor/read-path fast paths for compact projections, broader join/index coverage, grouped aggregates, and correlated subquery filters +
- Batch-first SQL row-batch execution foundation with batch-aware scan/index/join roots, shared predicate/projection kernels, and batch-native generic aggregate paths +
- Interactive CLI with meta-commands and file execution +
- REST API with 34 endpoints and OpenAPI/Scalar documentation +
- Blazor Server admin dashboard +
- Integrated Admin Forms and Reports designers with runtime preview/entry, database-backed metadata persistence, and print-ready report output +
- B+tree delete rebalancing with underflow handling (borrow/merge + interior collapse path) +
- Reusable snapshot reader sessions for higher concurrent-read throughput +
- Comprehensive benchmark suite (micro, macro, stress, scaling, in-memory, shared-memory) +
- Binary direct-payload collection storage with direct hydration and field/path extraction +
- Collection path indexes: nested scalar, array-element, nested array-object, Guid, temporal, ordered text +
- Collection path query APIs:
FindByPathAsyncandFindByPathRangeAsync
+ - Source-generated typed collection fast path foundations: generated collection models/codecs/field descriptors, trim-safe
GetGeneratedCollectionAsync<T>(...), generator diagnostics, NativeAOT trim-smoke validation, and a dedicated sample
+ - Full-text search with tokenization, stemming, and relevance ranking +
- Hybrid storage mode with lazy-resident durable storage and gRPC tunable file-cache +
- Client-wide
BackupAsync/RestoreAsyncacross direct, HTTP, gRPC, CLI, and Admin
+ ReplaceAsyncfor index stores
+ - Package-driven ETL pipelines with validation, dry-run, execute/resume, persisted run history, and Admin visual designer support +
- Architecture Guide — How the engine is structured +
- Internals & Contributing — How to extend the engine +
- Deployment & Installation Plan — Cross-platform distribution via dotnet tool, Docker, Homebrew, winget, and install scripts +
- Multi-Writer Follow-Up Plan — Post-initial multi-writer roadmap, insert-path gaps, and release criteria for broader completion +
- Query And Durable Write Performance Plan — Combined optimizer phase-2 plus durable-write completion plan, shipped state, and remaining benchmark/future-work boundaries +
- Multilingual Text Support Plan — Build on existing Unicode text storage with case-insensitive matching, locale-aware sorting, and
COLLATEclause support for queries and index definitions
+ - Database Encryption Plan — Encrypted storage format, key management, migration, and managed-surface rollout +
- Storage Engine Guide — CSharpDB.Storage API reference: device, pager, B+tree, WAL, indexing, serialization, and catalog +
- Native FFI Tutorials — Python and Node.js examples using the NativeAOT shared library +
- User-Defined Functions Plan — C# library functions callable by the database, native plugin extensions, and WASM sandboxing +
- Pub/Sub Change Events Plan — Engine-level change events with channel-based delivery for real-time data subscriptions +
- Benchmark Suite — Performance data informing optimization priorities +
docs/sql-reference.md. For the shorter curated page, see SQL Reference.SQL Reference
+Complete reference for the SQL dialect supported by CSharpDB.
++
Data Types
+| Type | +Aliases | +Description | +
|---|---|---|
INTEGER |
+ INT |
+ 64-bit signed integer | +
REAL |
+ FLOAT, DOUBLE |
+ 64-bit IEEE 754 floating point | +
TEXT |
+ VARCHAR |
+ UTF-8 Unicode string | +
BLOB |
+ — | +Raw binary data | +
NULL |
+ — | +Explicit NULL value (any column unless constrained NOT NULL) |
+
CSharpDB uses a flexible type system. Arithmetic operators perform implicit coercion + between numeric types where needed.
++
Statements
+CREATE TABLE
+CREATE TABLE [IF NOT EXISTS] table_name (
+ column_name type [PRIMARY KEY] [IDENTITY | AUTOINCREMENT] [NOT NULL]
+ [COLLATE collation_name]
+ [REFERENCES other_table(column) [ON DELETE CASCADE | RESTRICT]],
+ ...
+ );
+ Constraints:
+| Constraint | +Scope | +Description | +
|---|---|---|
PRIMARY KEY |
+ Column | +Designates the row identity column (integer) | +
IDENTITY / AUTOINCREMENT |
+ Column | +Auto-incrementing integer primary key | +
NOT NULL |
+ Column | +Rejects NULL values on insert/update | +
COLLATE |
+ Column | +Sets collation for TEXT comparisons (see Collations) | +
REFERENCES |
+ Column | +Foreign key referencing another table's column | +
ON DELETE CASCADE |
+ Foreign key | +Deletes child rows when parent is deleted | +
ON DELETE RESTRICT |
+ Foreign key | +Prevents deletion of parent row while children exist | +
ALTER TABLE
+ALTER TABLE table_name ADD COLUMN column_name type [constraints];
+ ALTER TABLE table_name DROP COLUMN column_name;
+ ALTER TABLE table_name DROP CONSTRAINT constraint_name;
+ ALTER TABLE table_name RENAME TO new_name;
+ ALTER TABLE table_name RENAME COLUMN old_name TO new_name;
+ DROP TABLE
+DROP TABLE [IF EXISTS] table_name;
+ CREATE INDEX
+CREATE [UNIQUE] INDEX [IF NOT EXISTS] index_name
+ ON table_name (column1 [, column2, ...]);
+ DROP INDEX
+DROP INDEX [IF EXISTS] index_name;
+ CREATE VIEW
+CREATE VIEW [IF NOT EXISTS] view_name AS select_statement;
+ DROP VIEW
+DROP VIEW [IF EXISTS] view_name;
+ CREATE TRIGGER
+CREATE TRIGGER [IF NOT EXISTS] trigger_name
+ {BEFORE | AFTER} {INSERT | UPDATE | DELETE}
+ ON table_name
+ [FOR EACH ROW]
+ [WHEN condition]
+ BEGIN
+ statement1;
+ [statement2;]
+ ...
+ END;
+ Triggers can reference NEW and OLD row aliases in their body and WHEN condition:
-
+
DROP TRIGGER
+DROP TRIGGER [IF EXISTS] trigger_name;
+ ANALYZE
+ANALYZE table_name;
+ Collects per-column statistics (distinct count, min/max, frequency histograms, quantile + buckets) and index prefix statistics used by the query planner for cardinality estimation + and operator selection. See Query Execution Pipeline for + details on how statistics influence planning.
++
Data Manipulation
+INSERT
+INSERT INTO table_name [(column1, column2, ...)]
+ VALUES (value1, value2, ...);
+ Column list is optional when providing values for all columns in declaration order.
+UPDATE
+UPDATE table_name
+ SET column1 = expression1 [, column2 = expression2, ...]
+ [WHERE condition];
+ DELETE
+DELETE FROM table_name
+ [WHERE condition];
+ SELECT
+SELECT [DISTINCT] column_list
+ FROM table_reference
+ [JOIN ...]
+ [WHERE condition]
+ [GROUP BY column1 [, column2, ...]]
+ [HAVING condition]
+ [ORDER BY column1 [ASC | DESC] [, ...]]
+ [LIMIT count]
+ [OFFSET skip];
+ Column List
+SELECT * -- all columns
+ SELECT column_name -- single column
+ SELECT column_name AS alias -- aliased column
+ SELECT table.column_name -- qualified column
+ SELECT expression -- computed value
+ SELECT aggregate_function(...) -- aggregate
+ FROM and JOIN
+FROM table_name [AS alias]
+
+ -- Join types
+ INNER JOIN table_name ON condition
+ LEFT JOIN table_name ON condition
+ RIGHT JOIN table_name ON condition
+ CROSS JOIN table_name
+ All join types except CROSS JOIN require an ON condition.
Subqueries
+-- Scalar subquery (must return a single value)
+ SELECT (SELECT MAX(age) FROM users) AS max_age;
+
+ -- IN subquery
+ WHERE column IN (SELECT id FROM other_table)
+ WHERE column NOT IN (SELECT id FROM other_table)
+
+ -- EXISTS subquery
+ WHERE EXISTS (SELECT 1 FROM other_table WHERE condition)
+ +
Common Table Expressions (CTEs)
+WITH cte_name [(column1, column2, ...)] AS (
+ select_statement
+ )
+ [, another_cte AS (...)]
+ SELECT ... FROM cte_name ...;
+ Multiple CTEs can be chained with commas. Optional column name lists rename the CTE's + output columns.
+++Note: The
+RECURSIVEkeyword is parsed but recursive CTE execution is not yet + implemented.
+
Set Operations
+select_statement UNION select_statement
+ select_statement INTERSECT select_statement
+ select_statement EXCEPT select_statement
+ Compound queries support trailing ORDER BY, LIMIT, and OFFSET applied to the
+ combined result.
+
Expressions and Operators
+Arithmetic
+| Operator | +Description | +
|---|---|
+ |
+ Addition | +
- |
+ Subtraction (also unary negation) | +
* |
+ Multiplication | +
/ |
+ Division (error on division by zero) | +
Comparison
+| Operator | +Description | +
|---|---|
= |
+ Equal | +
<> or != |
+ Not equal | +
< |
+ Less than | +
> |
+ Greater than | +
<= |
+ Less than or equal | +
>= |
+ Greater than or equal | +
Logical
+| Operator | +Description | +
|---|---|
AND |
+ Logical conjunction | +
OR |
+ Logical disjunction | +
NOT |
+ Logical negation | +
Special Expressions
+| Expression | +Example | +
|---|---|
BETWEEN ... AND ... |
+ WHERE age BETWEEN 18 AND 65 |
+
IN (...) |
+ WHERE status IN ('active', 'pending') |
+
NOT IN (...) |
+ WHERE id NOT IN (1, 2, 3) |
+
LIKE |
+ WHERE name LIKE 'J%' |
+
LIKE ... ESCAPE |
+ WHERE code LIKE '100\%%' ESCAPE '\' |
+
IS NULL |
+ WHERE email IS NULL |
+
IS NOT NULL |
+ WHERE email IS NOT NULL |
+
LIKE wildcards:
+| Wildcard | +Matches | +
|---|---|
% |
+ Zero or more characters | +
_ |
+ Exactly one character | +
+
Functions
+Aggregate Functions
+Used with or without GROUP BY. All except COUNT(*) ignore NULL values.
| Function | +Description | +Supports DISTINCT | +
|---|---|---|
COUNT(*) |
+ Number of rows | +— | +
COUNT(expr) |
+ Number of non-NULL values | +Yes | +
SUM(expr) |
+ Sum of numeric values | +Yes | +
AVG(expr) |
+ Average of numeric values | +Yes | +
MIN(expr) |
+ Minimum value | +— | +
MAX(expr) |
+ Maximum value | +— | +
SELECT COUNT(DISTINCT status), AVG(age) FROM users;
+ Scalar Functions
+| Function | +Arguments | +Returns | +Description | +
|---|---|---|---|
TEXT(expr) |
+ 1 | +TEXT | +Converts any value to its text representation | +
+
Parameters
+Named parameters are supported in value positions using the @ prefix:
SELECT * FROM users WHERE name = @name AND age > @minAge;
+ INSERT INTO users (name, age) VALUES (@name, @age);
+ UPDATE users SET name = @name WHERE id = @id;
+ DELETE FROM users WHERE id = @id;
+ Parameters cannot be used in identifier positions (table names, column names).
++
Collations
+Collations control how TEXT values are compared and sorted. They can be specified at the
+ column level in CREATE TABLE or at the expression level using the COLLATE operator.
| Collation | +Description | +
|---|---|
BINARY |
+ Byte-for-byte comparison (default) | +
NOCASE |
+ Case-insensitive comparison | +
NOCASE_AI |
+ Case-insensitive and accent-insensitive comparison | +
ICU:<locale> |
+ Unicode ICU-based comparison with locale support | +
-- Column-level collation
+ CREATE TABLE products (
+ name TEXT COLLATE NOCASE
+ );
+
+ -- Expression-level collation
+ SELECT * FROM products ORDER BY name COLLATE NOCASE_AI;
+ +
Limitations
+The following SQL features are not currently supported:
+-
+
SQL Reference
CSharpDB includes a full SQL engine with tokenizer, parser, query planner, and expression evaluator.
+Data Definition (DDL)
diff --git a/www/docs/storage-engine-reference.html b/www/docs/storage-engine-reference.html new file mode 100644 index 00000000..acf3752e --- /dev/null +++ b/www/docs/storage-engine-reference.html @@ -0,0 +1,1388 @@ + + + + + + + +docs/storage/README.md. For the shorter curated page, see Storage Engine Reference.CSharpDB.Storage
+A low-level, high-performance storage engine for .NET 10 built on top of RandomAccess and SafeFileHandle. It provides random-access async I/O, page caching, write-ahead logging (WAL), crash recovery, and the B+tree/index primitives that power the SQL engine and collection API.
For the guided storage tutorial track, including the new advanced standalone examples, start with samples/storage-tutorials/README.md. For the package-oriented storage API overview and tuning presets, see src/CSharpDB.Storage/README.md.
++
Table of Contents
+-
+
+
Architecture Overview
+┌──────────────────────────────────────────────────────┐
+ │ Application │
+ │ (SQL Engine / Collection API) │
+ ├──────────────────────────────────────────────────────┤
+ │ SchemaCatalog │
+ │ Tables ─ Indexes ─ Views ─ Triggers │
+ ├──────────────┬───────────────┬───────────────────────┤
+ │ BTree │ IndexStore │ RecordEncoder │
+ │ (data) │ (secondary) │ (row format) │
+ ├──────────────┴───────────────┴───────────────────────┤
+ │ Pager │
+ │ PageCache ─ DirtyTracking ─ PageAllocator │
+ ├──────────────────────┬───────────────────────────────┤
+ │ WriteAheadLog │ CheckpointCoordinator │
+ │ (WAL + WalIndex) │ (policy-driven) │
+ ├──────────────────────┴───────────────────────────────┤
+ │ IStorageDevice │
+ │ (FileStorageDevice / memory) │
+ └──────────────────────────────────────────────────────┘
+ Current storage builds can also enable a few newer read-path behaviors on top of this core layout:
+-
+
Page Layout:
+Page 0 (File Header):
+ [Magic: 4 bytes "CSDB"]
+ [FormatVersion: 4 bytes]
+ [PageSize: 4 bytes = 4096]
+ [PageCount: 4 bytes]
+ [SchemaRootPage: 4 bytes]
+ [FreelistHead: 4 bytes]
+ [ChangeCounter: 4 bytes]
+ [... reserved to 100 bytes ...]
+ [Slotted page content: 3996 bytes]
+
+ Pages 1+:
+ [SlottedPage: 4096 bytes]
+ [Header: 9 bytes]
+ PageType (1) ─ CellCount (2) ─ CellContentStart (2) ─ RightChild/NextLeaf (4)
+ [CellPointers: 2 bytes each]
+ [Free space]
+ [Cells: growing backward from page end]
+ +
FileStorageDevice
+FileStorageDevice wraps a SafeFileHandle opened with FileOptions.Asynchronous | FileOptions.RandomAccess, giving you:
-
+
public FileStorageDevice(string filePath, bool createNew = false)
+ | Parameter | +Description | +
|---|---|
filePath |
+ Path to the database file. | +
createNew |
+ true -> FileMode.CreateNew (fails if file exists). false -> OpenOrCreate. |
+
+
IStorageDevice Interface
+All storage operations go through IStorageDevice, making it easy to swap implementations (e.g., for in-memory testing).
public interface IStorageDevice : IAsyncDisposable, IDisposable
+ {
+ long Length { get; }
+ ValueTask<int> ReadAsync(long offset, Memory<byte> buffer, CancellationToken ct = default);
+ ValueTask WriteAsync(long offset, ReadOnlyMemory<byte> buffer, CancellationToken ct = default);
+ ValueTask FlushAsync(CancellationToken ct = default);
+ ValueTask SetLengthAsync(long length, CancellationToken ct = default);
+ }
+ +
Device Scenarios
+1. Create a New File
+Creates the file, failing if it already exists.
+await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+ Console.WriteLine($"File created. Length: {device.Length}"); // 0
+ 2. Open an Existing File
+Opens the file if it exists, or creates it if it does not.
+await using var device = new FileStorageDevice("mydb.cdb");
+ Console.WriteLine($"Opened. Length: {device.Length}");
+ 3. Write Raw Bytes at an Offset
+Writes are position-independent. Multiple writes at different offsets can be issued concurrently without locking.
+await using var device = new FileStorageDevice("mydb.cdb");
+ byte[] payload = "Hello, CSharpDB!"u8.ToArray();
+ await device.WriteAsync(offset: 0, payload);
+ 4. Read Raw Bytes from an Offset
+ReadAsync loops internally until the buffer is fully filled or EOF is reached.
await using var device = new FileStorageDevice("mydb.cdb");
+ var buffer = new byte[16];
+ int bytesRead = await device.ReadAsync(offset: 0, buffer);
+ Console.WriteLine($"Read {bytesRead} byte(s): {System.Text.Encoding.UTF8.GetString(buffer, 0, bytesRead)}");
+ 5. Read Past End of File (Zero-Fill Behavior)
+If you read a range that extends beyond the current file length, ReadAsync zero-fills the remainder of the buffer and returns the number of bytes that were actually on disk. This is useful for treating uninitialized pages as zeroed memory.
await using var device = new FileStorageDevice("mydb.cdb");
+ // File is currently 16 bytes; request 4096 bytes
+ var buffer = new byte[4096];
+ int bytesRead = await device.ReadAsync(offset: 0, buffer);
+ Console.WriteLine($"Bytes on disk: {bytesRead}"); // 16
+ Console.WriteLine($"Remainder is zeros: {buffer[16] == 0}"); // true
+ 6. Pre-allocate / Extend File Length
+Pre-allocating avoids fragmentation on spinning disks and is required before writing pages beyond the current end-of-file on some file systems.
+await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+ const int PageSize = 4096;
+ const int InitialPages = 8;
+ await device.SetLengthAsync(PageSize * InitialPages);
+ Console.WriteLine($"Pre-allocated: {device.Length} bytes"); // 32768
+ 7. Flush to Disk (fsync)
+FlushAsync calls RandomAccess.FlushToDisk, which issues a full fsync/FlushFileBuffers. Call this after committing a transaction to guarantee durability.
await using var device = new FileStorageDevice("mydb.cdb");
+ byte[] data = new byte[4096];
+ await device.WriteAsync(offset: 0, data);
+ await device.FlushAsync(); // durable on disk after this returns
+ 8. Check File Length
+The Length property reads the current file size directly from the OS without seeking.
await using var device = new FileStorageDevice("mydb.cdb");
+ long pages = device.Length / 4096;
+ Console.WriteLine($"Database has {pages} page(s).");
+ 9. Dispose Synchronously
+FileStorageDevice implements IDisposable for non-async contexts such as unit tests or top-level using statements.
using var device = new FileStorageDevice("mydb.cdb");
+ // ... operations ...
+ // Disposed when leaving the scope.
+ 10. Dispose Asynchronously
+Prefer await using in async code paths to align with the async I/O model.
await using var device = new FileStorageDevice("mydb.cdb");
+ // ... operations ...
+ // DisposeAsync called automatically.
+ 11. Cancellation Support
+All async methods accept a CancellationToken. Pass one to abort long reads or writes cleanly.
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
+ await using var device = new FileStorageDevice("mydb.cdb");
+ var buffer = new byte[4096];
+ try
+ {
+ await device.ReadAsync(offset: 0, buffer, cts.Token);
+ }
+ catch (OperationCanceledException)
+ {
+ Console.WriteLine("Read was cancelled.");
+ }
+ 12. Injecting via IStorageDevice (Testability)
+Program against IStorageDevice so you can substitute a fake or in-memory implementation in tests without touching the file system.
// Production wiring
+ IStorageDevice device = new FileStorageDevice("mydb.cdb");
+ var pager = await Pager.CreateAsync(device, wal, walIndex);
+
+ // In a unit test -- swap in your own IStorageDevice implementation
+ public sealed class MemoryStorageDevice : IStorageDevice
+ {
+ private byte[] _data = [];
+ public long Length => _data.Length;
+
+ public ValueTask<int> ReadAsync(long offset, Memory<byte> buffer, CancellationToken ct = default)
+ {
+ int available = (int)Math.Max(0, _data.Length - offset);
+ int toCopy = Math.Min(buffer.Length, available);
+ _data.AsMemory((int)offset, toCopy).CopyTo(buffer);
+ buffer[toCopy..].Span.Clear();
+ return ValueTask.FromResult(toCopy);
+ }
+
+ public ValueTask WriteAsync(long offset, ReadOnlyMemory<byte> buffer, CancellationToken ct = default)
+ {
+ long needed = offset + buffer.Length;
+ if (needed > _data.Length)
+ Array.Resize(ref _data, (int)needed);
+ buffer.CopyTo(_data.AsMemory((int)offset));
+ return ValueTask.CompletedTask;
+ }
+
+ public ValueTask FlushAsync(CancellationToken ct = default) => ValueTask.CompletedTask;
+
+ public ValueTask SetLengthAsync(long length, CancellationToken ct = default)
+ {
+ Array.Resize(ref _data, (int)length);
+ return ValueTask.CompletedTask;
+ }
+
+ public ValueTask DisposeAsync() => ValueTask.CompletedTask;
+ public void Dispose() { }
+ }
+ 13. Writing Fixed-Size Pages (4 KB)
+The storage engine works in 4 096-byte pages (see PageConstants.PageSize). Write a page at a computed offset.
await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+ const int PageSize = 4096;
+
+ // Pre-allocate space for 4 pages
+ await device.SetLengthAsync(PageSize * 4);
+
+ // Build a page payload
+ byte[] page = new byte[PageSize];
+ System.Text.Encoding.UTF8.GetBytes("page-0 data").CopyTo(page.AsSpan());
+
+ // Write page 0 at offset 0, page 1 at offset 4096, etc.
+ uint pageId = 0;
+ await device.WriteAsync(offset: (long)pageId * PageSize, page);
+ 14. Reading Fixed-Size Pages (4 KB)
+Read a page back by computing its byte offset from its page number.
+await using var device = new FileStorageDevice("mydb.cdb");
+ const int PageSize = 4096;
+
+ uint pageId = 0;
+ byte[] page = new byte[PageSize];
+ int read = await device.ReadAsync(offset: (long)pageId * PageSize, page);
+ Console.WriteLine($"Page {pageId}: read={read}, first bytes={System.Text.Encoding.UTF8.GetString(page, 0, 10)}");
+ 15. Appending Sequential Pages
+Grow the file by one page at a time and write content into each new page.
+await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+ const int PageSize = 4096;
+ int pagesToAppend = 3;
+
+ for (int i = 0; i < pagesToAppend; i++)
+ {
+ long newLength = device.Length + PageSize;
+ await device.SetLengthAsync(newLength);
+
+ byte[] page = new byte[PageSize];
+ BitConverter.TryWriteBytes(page, i); // store page index in first 4 bytes
+ await device.WriteAsync(offset: newLength - PageSize, page);
+ }
+
+ Console.WriteLine($"Final file size: {device.Length} bytes"); // 12288
+ await device.FlushAsync();
+ +
Pager
+The Pager sits between the B+tree layer and the storage device. It owns the page cache, tracks dirty pages, coordinates transactions, manages WAL integration, and drives checkpointing.
public static async ValueTask<Pager> CreateAsync(
+ IStorageDevice device,
+ IWriteAheadLog wal,
+ WalIndex walIndex,
+ PagerOptions? options = null,
+ CancellationToken ct = default)
+ P1. Create a New Database
+await using var device = new FileStorageDevice("mydb.cdb", createNew: true);
+ var walIndex = new WalIndex();
+ await using var wal = new WriteAheadLog("mydb.cdb", walIndex);
+ await wal.OpenAsync(currentDbPageCount: 0);
+
+ var pager = await Pager.CreateAsync(device, wal, walIndex);
+ await pager.InitializeNewDatabaseAsync(); // writes file header (page 0)
+
+ Console.WriteLine($"Pages: {pager.PageCount}"); // 1 (the file header page)
+ P2. Open and Recover an Existing Database
+On startup, call RecoverAsync to redo any committed WAL frames that were not yet checkpointed.
await using var device = new FileStorageDevice("mydb.cdb");
+ var walIndex = new WalIndex();
+ await using var wal = new WriteAheadLog("mydb.cdb", walIndex);
+
+ var pager = await Pager.CreateAsync(device, wal, walIndex);
+ await pager.RecoverAsync(); // replays committed WAL frames
+ P3. Read and Write Pages
+// Read a page (checks cache -> WAL -> device)
+ byte[] page = await pager.GetPageAsync(pageId: 1);
+
+ // Modify the page buffer in-place, then mark dirty
+ page[0] = 0xFF;
+ await pager.MarkDirtyAsync(pageId: 1); // tracked for WAL write on commit
+ P4. Allocate and Free Pages
+// Allocate a new page (extends file or reuses from freelist)
+ uint newPageId = await pager.AllocatePageAsync();
+
+ // Free a page (adds to freelist for reuse)
+ await pager.FreePageAsync(newPageId);
+ P5. Transaction Lifecycle
+Single writer per database. Reads do not require transactions.
+await pager.BeginTransactionAsync();
+ try
+ {
+ // ... modify pages via B+tree ...
+ await pager.CommitAsync(); // writes dirty pages to WAL, fsync
+ }
+ catch
+ {
+ await pager.RollbackAsync(); // discards uncommitted WAL frames
+ throw;
+ }
+ P6. Snapshot Isolation (Concurrent Readers)
+Multiple readers can run concurrently with a single writer. Each reader sees a consistent snapshot.
+// Writer thread: acquire snapshot for a reader
+ WalSnapshot snapshot = pager.AcquireReaderSnapshot();
+
+ // Reader thread: create a snapshot pager that sees only committed data
+ Pager snapshotPager = pager.CreateSnapshotReader(snapshot);
+ byte[] page = await snapshotPager.GetPageAsync(pageId: 1);
+
+ // When reader is done:
+ pager.ReleaseReaderSnapshot();
+ P7. Manual Checkpoint
+Copy all committed WAL frames to the main database file, then reset the WAL. This invalidates transient WAL-backed reads, and can also invalidate transient memory-mapped views; with PreserveOwnedPagesOnCheckpoint = true, already-owned main-file pages can remain resident in the shared cache.
await pager.CheckpointAsync();
+ P8. Configure Checkpoint Policy
+var options = new PagerOptions
+ {
+ CheckpointPolicy = new AnyCheckpointPolicy(
+ new FrameCountCheckpointPolicy(threshold: 500),
+ new TimeIntervalCheckpointPolicy(TimeSpan.FromMinutes(5))
+ ),
+ AutoCheckpointExecutionMode = AutoCheckpointExecutionMode.Background,
+ AutoCheckpointMaxPagesPerStep = 64
+ };
+
+ var pager = await Pager.CreateAsync(device, wal, walIndex, options);
+ // Auto-checkpoint triggers after 500 frames OR 5 minutes, whichever comes first.
+ // In background mode, the checkpoint runs after commit in smaller slices.
+ Built-in policies:
+| Policy | +Triggers When | +
|---|---|
FrameCountCheckpointPolicy(n) |
+ Committed frame count exceeds n |
+
WalSizeCheckpointPolicy(bytes) |
+ Estimated WAL size exceeds bytes |
+
TimeIntervalCheckpointPolicy(span) |
+ Elapsed time since last checkpoint exceeds span |
+
AnyCheckpointPolicy(...) |
+ Any sub-policy triggers | +
+
B+Tree
+B+tree keyed by signed 64-bit long keys. Leaf pages store (key, payload) pairs; interior pages store routing keys and child pointers. Supports forward-only cursor iteration, cache-only fast paths, and page-level rebalance/merge on delete.
B1. Create a New B+Tree
+// Allocates a root page and returns its ID
+ uint rootPageId = await BTree.CreateNewAsync(pager);
+
+ // Open the tree
+ var tree = new BTree(pager, rootPageId);
+ B2. Insert a Key-Value Pair
+Payload is raw bytes -- the B+tree has no opinion on format.
+byte[] payload = System.Text.Encoding.UTF8.GetBytes("Hello, B+tree!");
+ await tree.InsertAsync(key: 42, payload);
+ If the leaf page is full, the tree automatically splits the leaf and propagates the split key up to interior pages.
+B3. Point Lookup
+byte[]? result = await tree.FindAsync(key: 42);
+ if (result is not null)
+ Console.WriteLine(System.Text.Encoding.UTF8.GetString(result));
+ B4. Cache-Only Fast Path
+Avoids async I/O when all required pages are already cached.
+if (tree.TryFindCached(key: 42, out byte[]? payload))
+ {
+ // Hit: payload is definitive (null = not found, non-null = value)
+ Console.WriteLine($"Cache hit: {payload is not null}");
+ }
+ else
+ {
+ // Miss: need to call FindAsync for full traversal
+ payload = await tree.FindAsync(key: 42);
+ }
+ B5. Delete a Key
+bool deleted = await tree.DeleteAsync(key: 42);
+ Console.WriteLine(deleted ? "Deleted." : "Key not found.");
+ B6. Forward Cursor Scan
+Iterate all entries in key order. The cursor follows leaf-to-leaf next pointers (no interior page I/O after the first leaf).
+var cursor = tree.CreateCursor();
+ while (await cursor.MoveNextAsync())
+ {
+ long key = cursor.CurrentKey;
+ ReadOnlyMemory<byte> value = cursor.CurrentValue;
+ Console.WriteLine($"Key={key}, PayloadSize={value.Length}");
+ }
+ B7. Seek to a Specific Key
+Position the cursor at the first key >= target, then iterate forward.
+var cursor = tree.CreateCursor();
+ if (await cursor.SeekAsync(targetKey: 100))
+ {
+ do
+ {
+ Console.WriteLine($"Key={cursor.CurrentKey}");
+ } while (await cursor.MoveNextAsync());
+ }
+ B8. Count Entries
+Walks all leaf pages and sums cell counts.
+long count = await tree.CountEntriesAsync();
+ Console.WriteLine($"Tree contains {count} entries.");
+ +
Write-Ahead Log (WAL)
+Redo-style WAL for crash recovery and concurrent snapshot-isolated readers. Each commit writes dirty pages as frames to the WAL file. On checkpoint, committed frames are copied to the main database file.
+WAL File Format:
+ [WAL Header: 32 bytes]
+ Magic ─ Version ─ PageSize ─ Checksum salt
+ [Frame 0: 4120 bytes]
+ [FrameHeader: 24 bytes] ─ PageId ─ DbPageCount ─ Checksum
+ [PageData: 4096 bytes]
+ [Frame 1: 4120 bytes]
+ ...
+ W1. Open or Create a WAL
+var walIndex = new WalIndex();
+ await using var wal = new WriteAheadLog("mydb.cdb", walIndex);
+ await wal.OpenAsync(currentDbPageCount: pager.PageCount);
+ // WAL file: "mydb.cdb.wal"
+ W2. Write Transaction to WAL
+wal.BeginTransaction();
+
+ // Append modified pages as frames
+ await wal.AppendFrameAsync(pageId: 1, pageData);
+ await wal.AppendFrameAsync(pageId: 5, pageData);
+
+ // Commit: the last frame gets dbPageCount > 0, marking the commit boundary
+ await wal.CommitAsync(newDbPageCount: pager.PageCount);
+
+ // Or rollback: truncates uncommitted frames
+ await wal.RollbackAsync();
+ W3. Take a Reader Snapshot
+Freeze the WAL index at a point in time for a concurrent reader.
+WalSnapshot snapshot = walIndex.TakeSnapshot();
+
+ // Reader uses snapshot to resolve page lookups:
+ if (snapshot.TryGet(pageId: 1, out long walOffset))
+ {
+ byte[] page = await wal.ReadPageAsync(walOffset);
+ // page is the committed version at snapshot time
+ }
+ W4. Checkpoint WAL to Database File
+Copy all committed WAL pages to the main database file, then reset the WAL.
+await wal.CheckpointAsync(device, pageCount: pager.PageCount);
+ walIndex.Reset();
+ // WAL is now empty; all data is in the main file
+ +
Slotted Page Layout
+SlottedPage is a struct that overlays a byte[4096] buffer, providing structured access to variable-size cells within a fixed-size page.
[Header: 9 bytes]
+ PageType (1) ─ CellCount (2) ─ CellContentStart (2) ─ RightChild/NextLeaf (4)
+ [Cell Pointers: 2 bytes each, growing forward]
+ Offset to cell data for each cell
+ [Free Space]
+ [Cell Data: growing backward from page end]
+ Variable-size cells packed from the end
+ S1. Initialize a Page
+byte[] buffer = new byte[4096];
+ var sp = new SlottedPage(buffer, pageId: 1);
+ sp.Initialize(PageConstants.PageTypeLeaf);
+
+ Console.WriteLine($"Type: {sp.PageType}"); // Leaf
+ Console.WriteLine($"Cells: {sp.CellCount}"); // 0
+ Console.WriteLine($"Free: {sp.FreeSpace} bytes"); // ~4085
+ S2. Insert and Read Cells
+byte[] cellData = new byte[] { 0x01, 0x02, 0x03, 0x04 };
+ bool inserted = sp.InsertCell(index: 0, cellData);
+ Console.WriteLine($"Inserted: {inserted}"); // true
+
+ Span<byte> cell = sp.GetCell(index: 0);
+ Console.WriteLine($"Cell[0] length: {cell.Length}"); // 4
+ S3. Delete a Cell and Defragment
+sp.DeleteCell(index: 0);
+ Console.WriteLine($"Cells after delete: {sp.CellCount}"); // 0
+
+ // After many inserts/deletes, free space may be fragmented
+ sp.Defragment(); // rewrites cells contiguously at end of page
+ +
Indexing
+Secondary B+tree-backed indexes with optional caching and ordered range scan support.
+public interface IIndexStore
+ {
+ uint RootPageId { get; }
+ ValueTask<byte[]?> FindAsync(long key, CancellationToken ct = default);
+ ValueTask InsertAsync(long key, ReadOnlyMemory<byte> payload, CancellationToken ct = default);
+ ValueTask<bool> DeleteAsync(long key, CancellationToken ct = default);
+ IIndexCursor CreateCursor(IndexScanRange range);
+ }
+ I1. Create an Index Store
+uint indexRootPage = await BTree.CreateNewAsync(pager);
+ var indexTree = new BTree(pager, indexRootPage);
+ IIndexStore index = new BTreeIndexStore(indexTree);
+ I2. Insert and Lookup Index Entries
+Index payload typically contains the rowid(s) of matching rows.
+// Insert: key = hashed column value, payload = rowid (8 bytes)
+ byte[] rowIdPayload = BitConverter.GetBytes(42L);
+ await index.InsertAsync(key: hashOfColumnValue, rowIdPayload);
+
+ // Lookup
+ byte[]? result = await index.FindAsync(key: hashOfColumnValue);
+ if (result is not null)
+ {
+ long rowId = BitConverter.ToInt64(result);
+ Console.WriteLine($"Found rowid: {rowId}");
+ }
+ I3. Range Scan with Cursor
+var range = new IndexScanRange(
+ LowerBound: 100, LowerInclusive: true,
+ UpperBound: 200, UpperInclusive: false);
+
+ var cursor = index.CreateCursor(range);
+ while (await cursor.MoveNextAsync())
+ {
+ Console.WriteLine($"IndexKey={cursor.CurrentKey}");
+ }
+
+ // Full scan:
+ var fullCursor = index.CreateCursor(IndexScanRange.All);
+
+ // Point lookup as cursor:
+ var pointCursor = index.CreateCursor(IndexScanRange.At(42));
+ I4. Add Caching to an Index
+Wrap any IIndexStore with an LRU cache for repeated lookups.
IIndexStore cached = new CachingIndexStore(
+ inner: new BTreeIndexStore(indexTree),
+ capacity: 2048);
+
+ // Lookups check cache first; inserts/deletes update cache
+ byte[]? result = await cached.FindAsync(key: 42);
+ +
Record Serialization
+Compact binary encoding for database rows. Supports selective column projection and fast filter evaluation without materializing managed strings.
+Binary Format:
+ [columnCount: varint]
+ [col0_typeTag: 1 byte] [col0_data: ...]
+ [col1_typeTag: 1 byte] [col1_data: ...]
+ ...
+
+ Type Tags:
+ Null (0x00) -> no data
+ Integer (0x01) -> 8 bytes, little-endian long
+ Text (0x02) -> [length: varint] [UTF-8 bytes]
+ Real (0x03) -> 8 bytes, little-endian double (IEEE 754)
+ Blob (0x04) -> [length: varint] [raw bytes]
+ R1. Encode and Decode a Row
+var values = new DbValue[]
+ {
+ DbValue.FromInteger(1),
+ DbValue.FromText("Alice"),
+ DbValue.FromInteger(30)
+ };
+
+ byte[] encoded = RecordEncoder.Encode(values);
+ DbValue[] decoded = RecordEncoder.Decode(encoded);
+
+ Console.WriteLine($"Id={decoded[0].AsInteger}, Name={decoded[1].AsText}, Age={decoded[2].AsInteger}");
+ R2. Selective Column Projection
+Decode only the columns you need -- avoids materializing unused fields.
+// Decode only columns 0 and 1 (skip column 2)
+ DbValue[] partial = RecordEncoder.DecodeUpTo(encoded, maxColumnIndexInclusive: 1);
+
+ // Decode a single column by index
+ DbValue age = RecordEncoder.DecodeColumn(encoded, columnIndex: 2);
+ R3. Fast Filter Without Materialization
+Evaluate filters on encoded rows without allocating managed strings.
+// Check if column 1 equals "Alice" without creating a string
+ byte[] expectedUtf8 = "Alice"u8.ToArray();
+ if (RecordEncoder.TryColumnTextEquals(encoded, columnIndex: 1, expectedUtf8, out bool equals))
+ {
+ Console.WriteLine($"Column 1 is Alice: {equals}");
+ }
+
+ // Check numeric column for comparison
+ if (RecordEncoder.TryDecodeNumericColumn(encoded, columnIndex: 2,
+ out long intValue, out double realValue, out bool isReal))
+ {
+ Console.WriteLine($"Age: {intValue}");
+ }
+
+ // Check for null
+ bool isNull = RecordEncoder.IsColumnNull(encoded, columnIndex: 0);
+ R4. Varint Encoding
+Variable-length unsigned integer encoding (LEB128-style). Small values encode in 1 byte; up to 64-bit values supported.
+Span<byte> buffer = stackalloc byte[10];
+ int bytesWritten = Varint.Write(buffer, 300UL);
+
+ ulong value = Varint.Read(buffer, out int bytesRead);
+ Console.WriteLine($"Value: {value}, Bytes: {bytesRead}"); // 300, 2
+
+ int predictedSize = Varint.SizeOf(300UL); // 2
+ +
Schema Catalog
+B+tree-backed metadata store for tables, indexes, views, and triggers. Provides in-memory caching with a schema version counter for cache invalidation.
+C1. Initialize the Catalog
+var catalog = await SchemaCatalog.CreateAsync(pager);
+ Console.WriteLine($"Schema version: {catalog.SchemaVersion}");
+ C2. Create and Query Tables
+// Create a table
+ var schema = new TableSchema
+ {
+ TableName = "users",
+ Columns = new[]
+ {
+ new ColumnDefinition { Name = "id", Type = DbType.Integer, IsPrimaryKey = true },
+ new ColumnDefinition { Name = "name", Type = DbType.Text },
+ new ColumnDefinition { Name = "age", Type = DbType.Integer },
+ }
+ };
+ await catalog.CreateTableAsync(schema);
+
+ // Query table metadata
+ TableSchema? users = catalog.GetTable("users");
+ uint rootPage = catalog.GetTableRootPage("users");
+ BTree tableTree = catalog.GetTableTree("users");
+
+ // List all tables
+ IReadOnlyCollection<string> tableNames = catalog.GetTableNames();
+
+ // For snapshot readers
+ BTree snapshotTree = catalog.GetTableTree("users", snapshotPager);
+ C3. Create and Query Indexes
+var indexSchema = new IndexSchema
+ {
+ IndexName = "idx_users_name",
+ TableName = "users",
+ Columns = new[] { "name" },
+ IsUnique = false,
+ };
+ await catalog.CreateIndexAsync(indexSchema);
+
+ // Get index store
+ IIndexStore indexStore = catalog.GetIndexStore("idx_users_name");
+
+ // List indexes for a table
+ IReadOnlyList<IndexSchema> indexes = catalog.GetIndexesForTable("users");
+
+ // For snapshot readers
+ IIndexStore snapshotIndex = catalog.GetIndexStore("idx_users_name", snapshotPager);
+ C4. Views and Triggers
+// Views
+ await catalog.CreateViewAsync("active_users", "SELECT * FROM users WHERE age > 18");
+ string? viewSql = catalog.GetViewSql("active_users");
+ bool isView = catalog.IsView("active_users");
+
+ // Triggers
+ var trigger = new TriggerSchema
+ {
+ TriggerName = "trg_users_audit",
+ TableName = "users",
+ Event = TriggerEvent.AfterInsert,
+ Body = "INSERT INTO audit_log (table_name, action) VALUES ('users', 'INSERT')",
+ };
+ await catalog.CreateTriggerAsync(trigger);
+
+ IReadOnlyList<TriggerSchema> triggers = catalog.GetTriggersForTable("users");
+ C5. Persist Root Page Changes
+After B+tree operations that split the root page, persist the new root page ID in the catalog.
+// Persist for a single table + its indexes (fast)
+ await catalog.PersistRootPageChangesAsync("users");
+
+ // Persist for all tables and indexes (slower, used during batch operations)
+ await catalog.PersistAllRootPageChangesAsync();
+ +
Folder & File Storage
+FileStorageDevice is a raw byte device. To build a folder/file storage system on top of it, use the higher-level Database + Collection<T> API from CSharpDB.Engine. A single .cdb file (backed by one FileStorageDevice) holds all folders and files as typed collection documents in B+tree-backed collections.
++Add the Engine reference to your project if it is not already there:
++<ProjectReference Include="..\CSharpDB.Engine\CSharpDB.Engine.csproj" /> +
+
Domain Models
+Define records that represent a folder and a file entry. These are serialized as JSON by Collection<T>.
public record FolderEntry(
+ string Name,
+ string Path, // e.g. "/documents/reports"
+ DateTime CreatedAt,
+ string? Description = null);
+
+ public record FileEntry(
+ string Name,
+ string FolderPath, // parent folder path
+ string Content, // UTF-8 text content (or Base64 for binary)
+ string ContentType, // e.g. "text/plain", "application/json"
+ long SizeBytes,
+ DateTime CreatedAt,
+ DateTime UpdatedAt);
+ Keys follow a path convention:
+-
+
F1. Bootstrap the Storage
+Open (or create) a single .cdb file and obtain the two collections.
await using var db = await Database.OpenAsync("storage.cdb");
+ var folders = await db.GetCollectionAsync<FolderEntry>("folders");
+ var files = await db.GetCollectionAsync<FileEntry>("files");
+ Everything -- the B+tree pages, the WAL, and the page cache -- is managed by the single FileStorageDevice that Database.OpenAsync creates internally.
For a long-lived process that should keep touched pages hot while remaining durable on disk, use hybrid lazy-resident mode instead:
+await using var db = await Database.OpenHybridAsync(
+ "storage.cdb",
+ new DatabaseOptions(),
+ new HybridDatabaseOptions
+ {
+ PersistenceMode = HybridPersistenceMode.IncrementalDurable,
+ HotCollectionNames = ["folders", "files"]
+ });
+ That opens from disk lazily, keeps touched pages resident by cache policy, and can preload selected hot collections into the hybrid pager cache at startup.
+F2. Create a Folder
+Use the folder's path as the collection key so point lookups stay efficient; collection keys are hashed internally before probing the backing trees.
+async Task CreateFolderAsync(string path, string? description = null)
+ {
+ var entry = new FolderEntry(
+ Name: Path.GetFileName(path.TrimEnd('/')),
+ Path: path,
+ CreatedAt: DateTime.UtcNow,
+ Description: description);
+
+ await folders.PutAsync(path, entry);
+ }
+
+ await CreateFolderAsync("/documents");
+ await CreateFolderAsync("/documents/reports", description: "Monthly reports");
+ await CreateFolderAsync("/images");
+ F3. Create a File Inside a Folder
+The key is the full file path, which guarantees uniqueness across all folders.
+async Task CreateFileAsync(string folderPath, string fileName, string content, string contentType = "text/plain")
+ {
+ string key = $"{folderPath}/{fileName}";
+ var entry = new FileEntry(
+ Name: fileName,
+ FolderPath: folderPath,
+ Content: content,
+ ContentType: contentType,
+ SizeBytes: System.Text.Encoding.UTF8.GetByteCount(content),
+ CreatedAt: DateTime.UtcNow,
+ UpdatedAt: DateTime.UtcNow);
+
+ await files.PutAsync(key, entry);
+ }
+
+ await CreateFileAsync("/documents/reports", "q1.txt", "Q1 earnings: $1.2M");
+ await CreateFileAsync("/documents/reports", "q2.txt", "Q2 earnings: $1.5M");
+ await CreateFileAsync("/documents", "notes.md", "# Notes\nTodo list...", "text/markdown");
+ await CreateFileAsync("/images", "logo.svg", "<svg>...</svg>", "image/svg+xml");
+ F4. Read a File
+Retrieve a file by its full path key.
+FileEntry? file = await files.GetAsync("/documents/reports/q1.txt");
+
+ if (file is not null)
+ {
+ Console.WriteLine($"Name: {file.Name}");
+ Console.WriteLine($"Type: {file.ContentType}");
+ Console.WriteLine($"Size: {file.SizeBytes} bytes");
+ Console.WriteLine($"Content: {file.Content}");
+ }
+ else
+ {
+ Console.WriteLine("File not found.");
+ }
+ F5. List All Files in a Folder
+Collection<T>.FindAsync performs a full scan with an in-memory predicate -- suitable for small-to-medium collections.
string targetFolder = "/documents/reports";
+ await foreach (var kvp in files.FindAsync(f => f.FolderPath == targetFolder))
+ {
+ Console.WriteLine($" {kvp.Value.Name} ({kvp.Value.SizeBytes} bytes) [{kvp.Value.UpdatedAt:u}]");
+ }
+ F6. List All Folders
+await foreach (var kvp in folders.ScanAsync())
+ {
+ var f = kvp.Value;
+ Console.WriteLine($"{f.Path,-40} created {f.CreatedAt:u}");
+ }
+ F7. Update File Content
+PutAsync is an upsert -- it replaces the document at the key if it already exists.
async Task UpdateFileAsync(string filePath, string newContent)
+ {
+ FileEntry? existing = await files.GetAsync(filePath);
+ if (existing is null) throw new FileNotFoundException($"File not found: {filePath}");
+
+ var updated = existing with
+ {
+ Content = newContent,
+ SizeBytes = System.Text.Encoding.UTF8.GetByteCount(newContent),
+ UpdatedAt = DateTime.UtcNow
+ };
+
+ await files.PutAsync(filePath, updated);
+ }
+
+ await UpdateFileAsync("/documents/reports/q1.txt", "Q1 earnings: $1.4M (revised)");
+ F8. Delete a File
+bool deleted = await files.DeleteAsync("/documents/reports/q2.txt");
+ Console.WriteLine(deleted ? "File deleted." : "File not found.");
+ F9. Delete a Folder and Its Contents
+There is no cascading delete built in, so collect the child keys first, then delete in a single transaction.
+async Task DeleteFolderAsync(string folderPath)
+ {
+ // Collect all file keys under this folder
+ var toDelete = new List<string>();
+ await foreach (var kvp in files.FindAsync(f => f.FolderPath.StartsWith(folderPath, StringComparison.Ordinal)))
+ toDelete.Add(kvp.Key);
+
+ await db.BeginTransactionAsync();
+ try
+ {
+ foreach (var key in toDelete)
+ await files.DeleteAsync(key);
+ await folders.DeleteAsync(folderPath);
+ await db.CommitAsync();
+ }
+ catch
+ {
+ await db.RollbackAsync();
+ throw;
+ }
+ }
+
+ await DeleteFolderAsync("/documents/reports");
+ F10. Rename or Move a File
+CSharpDB does not have a rename primitive; copy the document to the new key and delete the old one inside a transaction.
+async Task MoveFileAsync(string sourcePath, string destinationPath)
+ {
+ FileEntry? source = await files.GetAsync(sourcePath);
+ if (source is null) throw new FileNotFoundException($"Source not found: {sourcePath}");
+
+ string newFolder = Path.GetDirectoryName(destinationPath)!.Replace('\\', '/');
+ string newFileName = Path.GetFileName(destinationPath);
+ var moved = source with
+ {
+ Name = newFileName,
+ FolderPath = newFolder,
+ UpdatedAt = DateTime.UtcNow
+ };
+
+ await db.BeginTransactionAsync();
+ try
+ {
+ await files.PutAsync(destinationPath, moved);
+ await files.DeleteAsync(sourcePath);
+ await db.CommitAsync();
+ }
+ catch
+ {
+ await db.RollbackAsync();
+ throw;
+ }
+ }
+
+ await MoveFileAsync("/documents/notes.md", "/documents/reports/notes.md");
+ F11. Search Files by Predicate
+Find all Markdown files larger than 100 bytes modified after a given date.
+DateTime since = new DateTime(2025, 1, 1, 0, 0, 0, DateTimeKind.Utc);
+ await foreach (var kvp in files.FindAsync(f =>
+ f.ContentType == "text/markdown" &&
+ f.SizeBytes > 100 &&
+ f.UpdatedAt > since))
+ {
+ Console.WriteLine($"{kvp.Key} ({kvp.Value.SizeBytes} bytes)");
+ }
+ F12. Bulk Create with an Explicit Transaction
+Wrap multiple writes in a single transaction so they all succeed or all roll back together.
+string[] reportNames = ["jan.txt", "feb.txt", "mar.txt", "apr.txt"];
+
+ await db.BeginTransactionAsync();
+ try
+ {
+ await CreateFolderAsync("/archive/2025");
+ foreach (var name in reportNames)
+ await CreateFileAsync("/archive/2025", name, $"Report: {name}");
+ await db.CommitAsync();
+ Console.WriteLine($"Committed {reportNames.Length} files in one transaction.");
+ }
+ catch
+ {
+ await db.RollbackAsync();
+ throw;
+ }
+ F13. SQL-Based Approach
+If you prefer a relational model, create folders and files tables with SQL and use ExecuteAsync.
await using var db = await Database.OpenAsync("storage.cdb");
+
+ // Create schema
+ await db.ExecuteAsync("""
+ CREATE TABLE IF NOT EXISTS folders (
+ id INTEGER PRIMARY KEY,
+ path TEXT NOT NULL,
+ name TEXT NOT NULL,
+ description TEXT,
+ created_at TEXT NOT NULL
+ )
+ """);
+
+ await db.ExecuteAsync("""
+ CREATE TABLE IF NOT EXISTS files (
+ id INTEGER PRIMARY KEY,
+ folder_path TEXT NOT NULL,
+ name TEXT NOT NULL,
+ content TEXT NOT NULL,
+ content_type TEXT NOT NULL,
+ size_bytes INTEGER NOT NULL,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL
+ )
+ """);
+
+ // Insert a folder
+ await db.ExecuteAsync("""
+ INSERT INTO folders (path, name, created_at)
+ VALUES ('/documents', 'documents', '2025-01-01T00:00:00Z')
+ """);
+
+ // Insert a file
+ await db.ExecuteAsync("""
+ INSERT INTO files (folder_path, name, content, content_type, size_bytes, created_at, updated_at)
+ VALUES ('/documents', 'readme.txt', 'Hello world', 'text/plain', 11,
+ '2025-01-01T00:00:00Z', '2025-01-01T00:00:00Z')
+ """);
+
+ // Query files in a folder
+ var result = await db.ExecuteAsync("SELECT name, size_bytes FROM files WHERE folder_path = '/documents'");
+ foreach (var row in result.Rows)
+ Console.WriteLine($"{row[0]} ({row[1]} bytes)");
+ F14. One Database File per Folder (Multi-Volume)
+Map each top-level folder to its own .cdb file. Each file gets its own FileStorageDevice instance, giving you independent WAL, checkpoint, and locking per folder.
// Each folder is a separate database file
+ var volumes = new Dictionary<string, Database>(StringComparer.Ordinal);
+
+ async ValueTask<Database> GetVolumeAsync(string folderName)
+ {
+ if (!volumes.TryGetValue(folderName, out var db))
+ {
+ db = await Database.OpenAsync($"{folderName}.cdb");
+ volumes[folderName] = db;
+ }
+ return db;
+ }
+
+ // Write to the "documents" volume
+ var docsDb = await GetVolumeAsync("documents");
+ var docsFiles = await docsDb.GetCollectionAsync<FileEntry>("files");
+ await docsFiles.PutAsync("readme.txt", new FileEntry(
+ Name: "readme.txt",
+ FolderPath: "/",
+ Content: "Welcome to the documents volume.",
+ ContentType: "text/plain",
+ SizeBytes: 32,
+ CreatedAt: DateTime.UtcNow,
+ UpdatedAt: DateTime.UtcNow));
+
+ // Write to the "images" volume
+ var imagesDb = await GetVolumeAsync("images");
+ var imagesFiles = await imagesDb.GetCollectionAsync<FileEntry>("files");
+ await imagesFiles.PutAsync("logo.svg", new FileEntry(
+ Name: "logo.svg",
+ FolderPath: "/",
+ Content: "<svg>...</svg>",
+ ContentType: "image/svg+xml",
+ SizeBytes: 14,
+ CreatedAt: DateTime.UtcNow,
+ UpdatedAt: DateTime.UtcNow));
+
+ // Dispose all volumes on shutdown
+ foreach (var (_, volume) in volumes)
+ await volume.DisposeAsync();
+ ++When to use multi-volume: large datasets where you want per-folder backup, different checkpoint intervals, or parallel writes to disjoint folders. For most use-cases a single
+.cdbfile is simpler.
+
Key Design Notes
+| Concern | +Detail | +
|---|---|
| No shared file pointer | +RandomAccess APIs are stateless w.r.t. position, so concurrent reads at different offsets are safe without locking. |
+
| Async-first | +All I/O is issued via RandomAccess.ReadAsync / WriteAsync, keeping the storage graph on the platform async file-I/O path. |
+
| Zero-fill on short reads | +ReadAsync always fills the entire buffer. Pages beyond EOF are returned as zeros, matching an uninitialized page convention used by the Pager. |
+
| fsync on flush | +FlushAsync calls RandomAccess.FlushToDisk which maps to FlushFileBuffers (Windows) or fsync (Linux/macOS), guaranteeing crash durability. |
+
| FileShare.Read | +Other processes can open the file read-only concurrently; write access is exclusive to the owning FileStorageDevice instance. |
+
| IDisposable + IAsyncDisposable | +Both patterns are supported; prefer await using in async code. |
+
| 4 KB page size | +All pages are PageConstants.PageSize (4096 bytes). Page 0 reserves 100 bytes for the file header. |
+
| Single writer, multiple readers | +The TransactionCoordinator enforces a single writer via SemaphoreSlim. Readers use WAL snapshots for isolation. |
+
| Optional memory-mapped reads | +The pager can use memory-mapped reads for clean main-file pages when PagerOptions.UseMemoryMappedReads is enabled and the storage device supports it. |
+
| Sequential scan read-ahead | +The pager can speculatively pull the next B+tree leaf page during forward scans when EnableSequentialLeafReadAhead is enabled. |
+
| Checkpoint residency preservation | +With PagerOptions.PreserveOwnedPagesOnCheckpoint, already-owned main-file pages can stay resident across checkpoint. This is what the engine's lazy-resident hybrid mode relies on. |
+
| B+tree leaf linking | +Leaf pages are linked via RightChildOrNextLeaf pointers, enabling efficient forward-only cursor scans without interior I/O. |
+
| Pluggable checkpoint policies | +ICheckpointPolicy allows frame-count, WAL-size, time-interval, or custom composite triggers. |
+
| Schema versioning | +SchemaCatalog.SchemaVersion increments on every DDL operation, enabling cache invalidation in upper layers. |
+
| Interceptor pipeline | +IPageOperationInterceptor provides hooks for diagnostics, metrics, and custom behavior on page reads, writes, transactions, and checkpoints. |
+
+
See Also
+-
+
On This Page
CSharpDB.Storage
A low-level, high-performance storage engine for .NET 10 built on top of RandomAccess and SafeFileHandle. It provides random-access async I/O, page caching, write-ahead logging (WAL), crash recovery, and the B+tree/index primitives that power the SQL engine and collection API.
Architecture Overview
docs/roadmap.md. For the shorter curated page, see Roadmap.CSharpDB Roadmap
+This document outlines the planned direction for CSharpDB, organized by timeframe and priority. Items are roughly ordered by expected impact within each tier, and statuses are intended to reflect the current v3.4.0 state of the repo.
+
Near-Term
+Recently completed improvements to query performance, storage/runtime behavior, maintenance workflows, and developer ergonomics.
+| Feature | +Description | +Status | +
|---|---|---|
DISTINCT keyword |
+ Deduplicate rows in SELECT output | +Done | +
| Composite indexes | +Multi-column indexes for covering more query patterns | +Done | +
| Index range scans | +Use indexes for <, >, <=, >=, BETWEEN — not just equality |
+ Done | +
| Prepared statement cache | +Cache parsed ASTs and query plans to avoid re-parsing identical SQL | +Done | +
| Cached max rowid | +Avoid repeated O(n) scans when generating row IDs on insert (in-memory + persisted high-water mark) | +Done | +
| B+tree delete rebalancing | +Merge underflowed pages on delete to reclaim space | +Done | +
| In-memory database mode | +Open a database fully in memory, load a disk database into memory, and save a committed snapshot back to disk | +Done | +
| Shared in-memory ADO.NET mode | +Support Data Source=:memory: and named shared in-memory databases with explicit save/load |
+ Done | +
| Collection field indexes | +Equality-based secondary indexes for Collection<T> via EnsureIndexAsync / FindByIndexAsync |
+ Done | +
| Reader session reuse | +Reuse snapshot pager and query planner inside ReaderSession for burst concurrent reads |
+ Done | +
| Architecture enforcement | +CSharpDB.Client is now the main caller-facing interaction layer across local and remote scenarios; ADO.NET now routes ordinary direct and daemon-backed access through that layer, with only named shared in-memory provider state still retaining an internal engine dependency |
+ Done | +
| Database administration | +Maintenance report, reindex (database/table/index/collection), VACUUM/compact, fragmentation analysis, database size report | +Done | +
| Dedicated gRPC daemon | +CSharpDB.Daemon host plus CSharpDB.Client gRPC coverage for SQL, schema, procedures, collections, and maintenance |
+ Done | +
| Storage tuning presets | +UseLookupOptimizedPreset() and UseWriteOptimizedPreset() for file-backed workloads |
+ Done | +
| Memory-mapped main-file reads | +Opt-in mapped clean-page reads plus copy-on-write materialization for mutable access on file-backed databases | +Done | +
| Background WAL checkpointing | +Incremental/sliced auto-checkpointing to move work off the triggering commit | +Done | +
| SQL executor/read-path fast paths | +Compact scan and indexed-range projections, broader join lookup/covered paths, grouped/composite index aggregates, correlated subquery filter fast paths, and lower row materialization overhead | +Done | +
| Table/index statistics | +ANALYZE command with persisted row counts, column NDV/min/max, stale tracking, and initial stats-guided index selection in the query planner | +Done | +
| Collection binary payloads | +Binary direct-payload format with faster hydration, direct field/path extraction, and richer path-based indexing | +Done | +
| Collection path indexes | +Nested scalar, array-element, nested array-object, Guid, temporal, and ordered text path indexes with FindByPathAsync / FindByPathRangeAsync |
+ Done | +
| Hybrid storage mode | +Lazy-resident durable storage with gRPC tunable file-cache configuration; Admin direct local hosting keeps a warm in-process database instance and uses hybrid incremental-durable options by default | +Done | +
| Client backup/restore | +BackupAsync / RestoreAsync as first-class ICSharpDbClient operations across direct, HTTP, gRPC, CLI, and Admin |
+ Done | +
| Older DB foreign-key retrofit migration | +Validate/apply maintenance workflow that rewrites existing child tables with persisted FK metadata across direct, HTTP, gRPC, CLI, and Admin | +Done | +
+
Mid-Term
+SQL feature parity, provider/tooling compatibility, and ecosystem expansion.
+| Feature | +Description | +Status | +
|---|---|---|
| User-defined functions | +Broader built-in scalar function registry (UPPER, ABS, COALESCE, etc.), user-registered C# functions, native plugin extensions | +Planned | +
| Subqueries | +Scalar subqueries, IN (SELECT ...), EXISTS (SELECT ...), including correlated evaluation in WHERE, non-aggregate projection, and UPDATE/DELETE expressions |
+ Done | +
UNION / INTERSECT / EXCEPT |
+ Set operations across SELECT results, including use in top-level queries, views, and CTE query bodies | +Done | +
| Window functions | +ROW_NUMBER(), RANK(), DENSE_RANK(), LEAD(), LAG() |
+ Planned | +
DEFAULT column values |
+ Allow default expressions in column definitions | +Planned | +
CHECK constraints |
+ Arbitrary expression-based constraints per column or per table | +Planned | +
| Foreign key constraints | +v1 support for single-column, column-level REFERENCES with optional ON DELETE CASCADE, plus sys.foreign_keys and metadata/tooling surfaces |
+ Done | +
| Remote host consolidation | +CSharpDB.Daemon now hosts the existing REST/HTTP /api surface and gRPC from one long-running process backed by the same warm daemon-hosted client; standalone CSharpDB.Api remains supported for REST-only hosting |
+ Done | +
| Remote host security | +Add built-in authentication, authorization, and transport-security options for remote HTTP and gRPC access, including API keys, protected admin endpoints, and TLS/mTLS deployment support | +Planned | +
| Daemon service packaging | +Package the existing CSharpDB.Daemon host as a persistent background service across systemd, Windows Service, and launchd |
+ Done | +
| Cross-platform deployment | +Self-contained daemon archives and install scripts ship for Windows, Linux, and macOS; dotnet tool, Docker, Homebrew, and winget distribution remain future work | +In Progress | +
| NuGet package | +Publish and maintain CSharpDB.Engine, CSharpDB.Data, CSharpDB.Client, and CSharpDB.Primitives as the primary NuGet packages |
+ Done | +
| Connection pooling | +Pool underlying direct embedded sessions behind CSharpDbConnection to amortize open/close cost |
+ Done | +
| Admin dashboard improvements | +Richer SQL editor UX, query history, deeper diagnostics, and integrated Forms/Reports tooling beyond the core schema/procedure/storage surface | +Done | +
| Visual query designer | +Classic Admin query builder with source canvas, join editing, design grid, SQL preview, and saved designer layouts | +Done | +
| ETL pipelines | +Built-in package-driven pipeline runtime with validation, dry-run, execute/resume flows, API/CLI/client coverage, run history, and Admin visual designer support | +Done | +
| VS Code extension | +Schema explorer, SQL editor with IntelliSense, data browser, table designer, storage diagnostics | +Done | +
ADO.NET GetSchema collections |
+ Implement DbConnection.GetSchema() for standard metadata collections (MetaDataCollections, Tables, Columns, Indexes, Views, ForeignKeys) to support ORMs and tooling that discover schema through ADO.NET |
+ Done | +
| Multilingual text support | +BINARY, NOCASE, NOCASE_AI, and built-in ICU:<locale> collation now work across SQL schema/query semantics, metadata surfaces, and collection path indexes; dedicated ordered SQL text index optimization remains planned |
+ Done | +
+
Long-Term
+Advanced features and fundamental architecture enhancements.
+| Feature | +Description | +Status | +
|---|---|---|
| Full-text search | +Inverted index with tokenization, stemming, and relevance ranking | +Done | +
| JSON path querying | +Query into JSON document fields in the Collection API (e.g., $.address.city) via FindByPathAsync / FindByPathRangeAsync |
+ Done | +
| Advanced collection storage path | +Binary direct-payload format with direct binary hydration, path-based field extraction, and richer expression/path indexes | +Done | +
| SQL batched row transport | +Internal row-batch transport serves as the batch-first SQL execution foundation across batch-capable result boundaries, scans, joins, and generic aggregates | +Done | +
| Source-generated collection fast path | +In progress: GetGeneratedCollectionAsync<T>(...), generated field descriptors/index bindings, analyzer-packaged collection model/codecs, trim/NativeAOT smoke coverage, and a dedicated sample are now in place while broader package ergonomics and remaining generator coverage continue |
+ In Progress | +
| Page-level compression | +Compress cell content within pages to reduce I/O and storage | +Planned | +
| At-rest encryption | +Encrypt database and WAL files with passphrase-based key management and explicit plaintext/encrypted migration/export paths | +Research | +
| Advanced cost-based query optimizer | +In progress: phase-2 stats-guided costing is now in place through internal equi-depth histograms, heavy hitters, composite-index prefix distinct-count summaries, skew-aware lookup/filter estimates, correlation-aware composite equality filters/joins, and bounded DP reordering for small inner-join chains; adaptive re-optimization and public histogram inspection remain future work | +In Progress | +
| Async I/O batching | +In progress: WAL frame-chunk writes, chunked checkpoint page copies, shared snapshot/export batching, and reusable B-tree copy utilities now cover the main storage and maintenance write paths; remaining auditing is outside the WAL hot path | +In Progress | +
| Low-latency durable writes | +Done in v2.9.0: advisory planner-stat persistence can stay deferred without weakening committed-row durability, and sys.table_stats.row_count_is_exact now makes exact versus estimated row-count semantics explicit to planner and COUNT(*) fast paths |
+ Done | +
| Group commit / deferred WAL flush | +Done in v2.9.0: opt-in UseDurableCommitBatchWindow(...) batches durable WAL flushes across contending in-process transactions and remains an expert measure-first knob rather than default behavior |
+ Done | +
| Initial multi-writer support | +Explicit WriteTransaction conflict-detected retry flow, shared auto-commit non-insert isolation, and opt-in ConcurrentWriteTransactions for shared implicit inserts |
+ Done | +
| Broader multi-writer insert optimization | +Improve hot insert fan-in, row-id reservation, and other high-contention patterns beyond the current initial multi-writer path | +Research | +
| Replication / change feed | +Stream committed changes for read replicas or event-driven architectures | +Research | +
| WebAssembly sandboxed UDFs | +Execute untrusted user-submitted functions in a WASM sandbox with resource limits (fuel, memory caps) via Wasmtime | +Research | +
+
Current Limitations
+These are known simplifications in the current implementation:
+| Area | +Limitation | +
|---|---|
| Functions | +Very limited scalar function surface today: built-in TEXT(expr) plus aggregate functions; no broader built-in function library or user-defined functions yet |
+
| Query | +Scalar/IN/EXISTS subqueries are supported, including correlated cases in WHERE, non-aggregate projection, and UPDATE/DELETE expressions; correlated subqueries are not yet supported in JOIN ON, GROUP BY, HAVING, ORDER BY, or aggregate projections |
+
| Query | +UNION, INTERSECT, and EXCEPT are supported; UNION ALL is not implemented yet |
+
| Query | +No window functions | +
| Schema | +No SQL DEFAULT column values or CHECK constraints yet. Foreign keys are currently v1 only: single-column, column-level REFERENCES with optional ON DELETE CASCADE; table-level/composite/deferred foreign keys and ON UPDATE actions are not implemented |
+
| Indexes | +Equality lookups support current INTEGER/TEXT indexes, but ordered range-scan pushdown is still limited to single-column INTEGER index paths |
+
| RowId | +Legacy table schemas without persisted high-water metadata may pay a one-time key scan on first insert | +
| Collections | +FindByIndexAsync supports declared field-equality lookups; FindByPathAsync and FindByPathRangeAsync support path-based queries on indexed paths; FindAsync remains a full scan for unindexed predicates |
+
| Networking | +CSharpDB.Daemon now hosts both REST and gRPC from one process; named pipes remain reserved but are not implemented end to end today |
+
| Security | +Remote HTTP and gRPC deployment still rely on external network controls or front-end TLS termination; built-in authentication, authorization, and TLS/mTLS support are still planned | +
| Text / Multilingual | +Text is stored as UTF-8 and supports all Unicode languages; default semantics remain ordinal, but opt-in BINARY, NOCASE, NOCASE_AI, and ICU:<locale> collation are implemented for SQL and collection indexes. Dedicated ordered SQL text index optimization remains planned |
+
| Concurrency | +The physical WAL commit path is still serialized at the storage boundary. Initial multi-writer support is shipped, but observed gains still depend on conflict shape and whether shared auto-commit INSERT is left on the default serialized path |
+
| Storage | +No page-level compression | +
| Storage | +No at-rest encryption for database/WAL files; on-disk storage is plaintext only | +
| Storage | +Memory-mapped reads are opt-in and currently apply only to clean main-file pages; WAL-backed reads still rely on the WAL/cache path | +
| Storage | +By default, durable auto-commit single-row writes still pay a physical WAL flush per commit; opt-in UseDurableCommitBatchWindow(...) can trade some commit latency for higher throughput across contending in-process writers, but default behavior remains per-commit durable |
+
| Query | +Phase-2 cost-based planning is largely in place: ANALYZE, sys.table_stats, sys.column_stats, internal histograms/heavy hitters/prefix stats, and bounded small-chain join reordering now feed join/access-path costing; remaining work is adaptive re-optimization and public histogram/diagnostic surfacing rather than missing core stats-guided costing |
+
| Query | +Internal row-batch transport is now the default scan-heavy execution foundation across batch-capable scans, joins, aggregates, and result boundaries; remaining work is broader kernel specialization and optional SIMD-style tuning rather than missing core batch coverage | +
+
Completed Milestones
+Major features already implemented:
+-
+
+
See Also
+-
+
Roadmap
-Planned direction for CSharpDB — organized by timeframe and priority. Reflects the current v3.0.0 state.
+Planned direction for CSharpDB — organized by timeframe and priority. Reflects the current v3.4.0 state.
+Foreign Key Constraints
Remote Host Consolidation
- Planned + DoneFold REST/HTTP into CSharpDB.Daemon so one server host serves REST, gRPC, and future transports from a shared warm Database instance.
+CSharpDB.Daemon now hosts the existing REST/HTTP /api surface and gRPC from one long-running process backed by the same warm daemon-hosted client. Standalone CSharpDB.Api remains supported for REST-only hosting.
Remote Host Security
Daemon Service Packaging
- Planned + DonePackage CSharpDB.Daemon as a persistent background service across systemd, Windows Service, and launchd.
+CSharpDB.Daemon can be packaged as a persistent background service across systemd, Windows Service, and launchd.
Cross-Platform Distribution
- Planned + In Progressdotnet tool, self-contained binaries, Docker, Homebrew, winget, and install scripts.
+Self-contained daemon archives and install scripts ship for Windows, Linux, and macOS; dotnet tool, Docker, Homebrew, and winget distribution remain future work.
Current Limitations
FindByIndexAsync supports declared field-equality lookups; FindByPathAsync and FindByPathRangeAsync support path-based queries on indexed paths; FindAsync remains a full scan for unindexed predicatesCSharpDB.Daemon now hosts both REST and gRPC from one process; named pipes remain reserved but are not implemented end to end todayBINARY, NOCASE, NOCASE_AI, and ICU:<locale> collation are implemented for SQL and collection indexes; dedicated ordered SQL text index optimization remains planned