Skip to content

Isolate d1 import sqlite3 runs from user config#1288

Merged
no-itsbackpack merged 1 commit into
mainfrom
d1-import-hermetic-sqlite3
Jul 9, 2026
Merged

Isolate d1 import sqlite3 runs from user config#1288
no-itsbackpack merged 1 commit into
mainfrom
d1-import-hermetic-sqlite3

Conversation

@no-itsbackpack

Copy link
Copy Markdown
Contributor

Summary

A user hit this during a D1 import:

Import failed: count sqlite staging rows: expected integer

The cause is that the sqlite3 shell loads ~/.sqliterc even for one-shot invocations. If that file turns on headers or column mode, the output of SELECT COUNT(*) is no longer a bare integer and the import verification fails. Reproduced locally: with .headers on in ~/.sqliterc, sqlite3 db "SELECT COUNT(*) FROM t" prints COUNT(*) before the number, which is exactly what broke the parse. Setting $HOME does not help because sqlite3 finds the rc file through the passwd database.

Changes

  • All sqlite3 invocations in the d1 import (row counts, fingerprints, sampling, chunked dump loading) now go through a helper that passes -batch -noheader -init /dev/null, so output is always plain list mode regardless of the user's .sqliterc.
  • Replaced fmt.Sscanf("%d") with strict strconv.ParseInt. Sscanf accepts trailing garbage ("3 rows" parses as 3) and its "expected integer" error carries no context, which is why this bug was hard to diagnose from the report. Parse errors now quote the actual output.
  • runSQLiteQuery no longer parses CombinedOutput. sqlite3 can exit 0 while printing warnings to stderr (WAL recovery, lock warnings), and those bytes were being mixed into output we split and compare. Stdout is parsed, stderr is reported only on failure.
  • Failed counts now include sqlite3's stderr instead of just "exit status 1".
  • SQL identifiers are quoted with proper SQL escaping (" doubled) instead of Go's %q, which produces invalid SQL for names containing quotes or backslashes.

Testing

  • Reproduced the original failure with a .sqliterc containing .headers on and .mode column, then confirmed the hermetic flags produce clean output with the same rc file in place.
  • go test ./internal/import/d1/ passes, go vet and gofmt clean.

The sqlite3 shell loads ~/.sqliterc even for one-shot invocations, so
settings like .headers on or .mode column change the output format and
break row count parsing during import verification. Invoke sqlite3 with
-batch -noheader -init /dev/null so output is always plain list mode.

Also harden the surrounding parsing: replace Sscanf with strict
strconv.ParseInt so garbage output fails loudly instead of parsing
partially, include the offending output and stderr in errors, stop
mixing stderr into parsed query output, and quote SQL identifiers
properly instead of using Go's %q.

Co-authored-by: Cursor <cursoragent@cursor.com>
@no-itsbackpack no-itsbackpack requested a review from a team as a code owner July 9, 2026 16:59
@no-itsbackpack no-itsbackpack merged commit 29e6cc4 into main Jul 9, 2026
4 checks passed
@no-itsbackpack no-itsbackpack deleted the d1-import-hermetic-sqlite3 branch July 9, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants