Skip to content

fix(export): stream database dump to prevent memory exhaustion and unsafe SQL usage#114

Open
AbhishekMauryaGEEK wants to merge 2 commits intoouterbase:mainfrom
AbhishekMauryaGEEK:clean/dump-fix
Open

fix(export): stream database dump to prevent memory exhaustion and unsafe SQL usage#114
AbhishekMauryaGEEK wants to merge 2 commits intoouterbase:mainfrom
AbhishekMauryaGEEK:clean/dump-fix

Conversation

@AbhishekMauryaGEEK
Copy link

Summary

The current /export/dump implementation loads the entire database into memory and constructs the dump as a single string. This approach leads to performance and reliability issues for moderately large databases.

This PR replaces the in-memory dump construction with a streaming approach and introduces safer query handling.


Problems Identified

1. Memory Exhaustion / DoS Risk

The existing implementation accumulates the entire dump in a single string (dumpContent) before returning a response.
For large datasets, this can:

  • exhaust worker memory
  • cause request failures
  • degrade service reliability

2. Unsafe SQL Construction

Table names are directly interpolated into SQL queries:

SELECT * FROM ${table};

While table names originate from sqlite_master, relying on raw interpolation is unsafe and makes the code fragile.


3. Invalid Dump Format

The dump begins with a binary SQLite header:

SQLite format 3\0

but is followed by SQL statements, resulting in a mixed and non-standard dump format.


Changes Made

  • Replaced in-memory string accumulation with a ReadableStream-based response

  • Introduced chunked data fetching using LIMIT and OFFSET

  • Added strict identifier validation for table names

  • Parameterized schema lookup query (name = ?)

  • Replaced dump format with valid SQL transaction-based output:

    • BEGIN TRANSACTION
    • INSERT INTO ...
    • COMMIT

Impact

  • Prevents memory exhaustion during exports
  • Improves reliability for medium-to-large datasets
  • Produces valid and restorable SQL dump files
  • Reduces risk from unsafe query construction

Scope

This PR intentionally limits changes to the existing /export/dump route to keep the fix focused and reviewable.
It does not introduce new infrastructure (e.g. R2, async jobs, alarms), and instead addresses core correctness and safety issues in the current implementation.


How to Test

  1. Trigger dump:
curl --location 'http://localhost:8787/export/dump' \
--header 'Authorization: Bearer <TOKEN>' \
--output dump.sql
  1. Verify:
  • Dump downloads successfully
  • File size increases progressively (streamed)
  • No worker crash for larger tables
  1. Validate dump:
  • Import into SQLite
  • Ensure schema and data are intact

Notes

This change is designed as a minimal, safe improvement over the current implementation while maintaining existing behavior and API surface.

@AbhishekMauryaGEEK
Copy link
Author

/claim #59

This PR fixes the current /export/dump implementation by:

  • streaming the dump instead of buffering the entire database in memory
  • fetching rows in chunks to reduce timeout/memory pressure
  • validating identifiers and parameterizing schema lookup
  • producing a valid SQL dump format (BEGIN TRANSACTION ... COMMIT)

Focused scope: this intentionally fixes the safety and correctness issues in the existing route without introducing new infrastructure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant