Skip to content

fix: add streaming DMMF binary command to bypass V8 and WASM limits#5761

Open
chris-tophers wants to merge 1 commit intoprisma:mainfrom
chris-tophers:fix/dmmf-binary-streaming
Open

fix: add streaming DMMF binary command to bypass V8 and WASM limits#5761
chris-tophers wants to merge 1 commit intoprisma:mainfrom
chris-tophers:fix/dmmf-binary-streaming

Conversation

@chris-tophers
Copy link
Copy Markdown
Contributor

@chris-tophers chris-tophers commented Feb 6, 2026

Summary

Alternative approach to #5757 for bypassing V8's string length limit on large DMMF.

Adds a get-dmmf subcommand to the prisma-fmt CLI binary that streams DMMF JSON directly to stdout via serde_json::to_writer(). Unlike the buffered WASM approach, this has no memory ceiling — the native binary streams incrementally with zero intermediate allocation.

Root Cause

dmmf_json_from_validated_schema() uses serde_json::to_string(), producing a single Rust String. When wasm-bindgen converts this to a JS string, V8 rejects it if the string exceeds 0x1fffffe8 characters (~536MB). The WASM buffered approach (#5757) works around this by returning Vec<u8> chunks, but WASM32 linear memory is capped at ~4GB, giving a practical ceiling of ~1.5-2GB DMMF.

The binary streaming approach eliminates both limits: serde_json::to_writer() writes JSON tokens incrementally to stdout with only 1x peak memory (the in-memory DMMF struct — no serialized buffer).

Changes

File Change
query-compiler/dmmf/src/lib.rs Add dmmf_json_to_writer() using serde_json::to_writer()
prisma-fmt/src/get_dmmf.rs Add get_dmmf_to_writer() that validates schema + streams
prisma-fmt/src/lib.rs Export get_dmmf_to_writer()
prisma-fmt/src/main.rs Add GetDmmf CLI variant: reads JSON params from stdin, streams DMMF to stdout

+47 lines, 4 files changed. No existing behavior modified. No new dependencies.

Comparison: WASM Buffered vs Binary Streaming

Aspect WASM Buffered (#5757) Binary Streaming (this PR)
Memory ceiling ~1.5-2GB (WASM32 limit) Unlimited
Peak memory 2x DMMF (struct + Vec<u8>) 1x (struct only, streams)
New binary shipped No Yes (prisma-fmt)
Rust complexity Medium (Mutex, chunks) Low (to_writer + CLI cmd)
Lines added (Rust) ~100 ~47
Works today Yes (WASM-only path) Needs prisma-fmt in npm or PRISMA_FMT_BINARY env var

Usage

# Pipe schema params to stdin, get DMMF JSON on stdout
echo '{"prismaSchema":"datasource db { ... }","noColor":true}' | prisma-fmt get-dmmf

The TypeScript companion PR detects the V8 error and spawns this binary as a fallback, stream-parsing stdout with @streamparser/json.

Companion PR

TypeScript fallback: prisma/prisma#29149 (adds binary streaming fallback in TypeScript)

Notes

  • The Prisma team can choose either approach (fix: add buffered DMMF API to bypass V8 string length limit #5757 or this), or combine both (WASM buffered for moderate sizes, binary streaming for extreme sizes)
  • prisma-fmt is already compiled as a native binary in CI — it just isn't shipped via npm today
  • The PRISMA_FMT_BINARY env var allows users to point to a custom-built binary immediately

Fixes: prisma/prisma#29111

Add a `get-dmmf` subcommand to the prisma-fmt CLI that streams DMMF JSON
directly to stdout via serde_json::to_writer(). This approach has no
memory ceiling — unlike WASM (limited to ~4GB linear memory), the native
binary can stream arbitrarily large DMMF with only 1x peak memory
(the in-memory DMMF struct, no serialized buffer).

Changes:
- dmmf crate: add dmmf_json_to_writer() using serde_json::to_writer()
- prisma-fmt: add get_dmmf_to_writer() that validates + streams
- prisma-fmt: export get_dmmf_to_writer() from lib.rs
- prisma-fmt: add GetDmmf CLI variant, reads stdin params, streams to stdout

Alternative approach to the buffered WASM API in prisma#5757.
Fixes: prisma/prisma#29111

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prisma 7.x WASM DMMF generation fails with "Cannot create a string longer than 0x1fffffe8 characters"

1 participant