Skip to content

feat(data/json): unify JSON escapers and add streaming NDJSON emitter#348

Merged
octalide merged 2 commits into
devfrom
feat/338-streaming-json-emitter
Jul 2, 2026
Merged

feat(data/json): unify JSON escapers and add streaming NDJSON emitter#348
octalide merged 2 commits into
devfrom
feat/338-streaming-json-emitter

Conversation

@octalide

@octalide octalide commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Closes #338.

What

Moves the streaming ASCII escaper out of the mach compiler's mach.cli.json and into std.data.json as the one JSON emission home, unifying it with the tree emitter's #337 structural escaper into a single escape core.

One escaper, explicit policy knob

Both emission surfaces now escape string bodies through one core (escape_unit) selected by EscapePolicy:

  • ESCAPE_VERBATIM (structural-utf8-verbatim) — escapes ", \, and the C0 controls only; every other byte, including UTF-8 lead/continuation bytes, passes through verbatim. The tree emitter emit() uses this, so valid UTF-8 round-trips as UTF-8.
  • ESCAPE_ENSURE_ASCII — as above for bytes < 0x80, but each byte >= 0x80 is UTF-8 decoded (full RFC 3629 validation) and emitted as a \uXXXX escape (astral code points as a surrogate pair); invalid UTF-8 emits the U+FFFD replacement escape, so every emitted byte is printable ASCII. The streaming NDJSON surface uses this.

The tree emitter now renders through a counting io.Writer (BufWriter) instead of a bespoke Emitter, so both surfaces share one sink and one escaper. emit() keeps its exact measure-when-buffer-short semantics (returns the required length even when the buffer is too small).

Streaming NDJSON surface (full flat + nested/array)

Ported verbatim from the current-dev mach.cli.json (which mach#1838 extended past the original flat surface): Object, Array, object_begin/object_end, the nesting primitives object_end_value/field_object_begin/field_array_begin/array_end/array_object_begin, and the members field_str/field_str_or_null/field_null/field_i64/field_bool, plus write_json_string. The Object.first/Array.first comma-state machine is ported unchanged. object_end closes via object_end_value and writes the sole NDJSON line terminator, so a whole nested object lands on one line. Public names match the compiler's exactly, so the mach-side rewire (refactor/338-json-to-std, rewiring both testing.mach and diagnostic.mach/flush_json) is a use swap with no call-site churn.

Placement rationale

Kept in std.data.json itself (not a sibling module). The one escape core stays private, shared by both public emission surfaces; a sibling module would force exporting the escaper or adding a third module just to share it. "One home" = one module.

Byte-identity

  • Tree emitter: the pre-existing exact-byte emit tests (null/number/bool, array/object round-trips, quote/backslash/control escapes, object keys) all still pass unchanged.
  • Streaming emitter: the exact-byte write_json_string tests were ported over from mach.cli.json (identical inputs and want byte arrays); mach#1838's compose tests (nested object + array-of-objects {"a":1,"loc":{"line":2,"col":3},"rel":[{"label":"x"},{"label":"y"}]}\n, and empty array {"rel":[]}\n) were ported verbatim; and a flat full-object sequence test asserts {"n":7,"s":"a\"b","b":true,"o":null,"z":null}\n byte-for-byte (exercising field_null).

Scope

  • #337/#340 parse/emit representation-asymmetry warnings preserved intact; #340 stays open.
  • CHANGELOG.md [Unreleased] entry added.

Verification

mach build . and mach test . green — 588 passed, 0 failed (36 in std.data.json: 23 tree + 13 streaming). Both CI checks (build + cross-riscv64) pass.

Note: #338 is a mach-std issue; this merge closes it on the mach-std side. The paired mach-side refactor (refactor/338-json-to-std) lands separately after the mach-std release and mach.lock bump.

🤖 Generated with Claude Code

Move the streaming ASCII escaper from the mach compiler's mach.cli.json
into std.data.json as the one JSON emission home, and unify it with the
tree emitter's structural escaper into a single escape core.

escape_unit takes an explicit policy: ESCAPE_VERBATIM (structural, UTF-8
bytes verbatim) drives the tree emitter emit(); ESCAPE_ENSURE_ASCII
(RFC 3629 decode to \uXXXX, astral surrogate pairs, U+FFFD on invalid
input) drives the streaming surface. The tree emitter now renders through
a counting io.Writer (BufWriter) so both surfaces share one sink and one
escaper; emit() keeps its measure-when-short semantics.

Add the streaming NDJSON surface (Object, object_begin/object_end,
field_str/field_str_or_null/field_i64/field_bool, write_json_string)
ported verbatim from mach.cli.json, with its exact-byte tests plus a
full-object call-sequence test. Tree and streaming output are
byte-identical to their pre-unification form. The #337/#340 parse/emit
representation asymmetry warnings are preserved.

Closes #338
@octalide octalide marked this pull request as ready for review July 2, 2026 21:10
…ters

Re-port the streaming NDJSON surface from the current mach dev, which
mach#1838 extended past the flat-object surface the first port captured.

Add the Array record and the nesting primitives object_end_value,
field_object_begin, field_array_begin, array_end, array_object_begin,
and field_null, ported verbatim with their Object.first / Array.first
comma-state machine intact. object_end now closes via object_end_value
and writes the sole NDJSON line terminator, so a whole nested object
still lands on one line.

Port mach#1838's exact-byte compose tests (nested object + array-of-
objects, empty array) and exercise field_null in the flat sequence test.
The full public surface and byte output match the compiler's current
mach.cli.json, so the mach-side rewire is a use swap.
@octalide octalide merged commit 4e26deb into dev Jul 2, 2026
2 checks passed
@octalide octalide deleted the feat/338-streaming-json-emitter branch July 2, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

data/json: adopt the streaming ASCII escaper as the one JSON emission home (unify with mach.cli.json)

1 participant