feat(data/json): unify JSON escapers and add streaming NDJSON emitter#348
Merged
Conversation
Move the streaming ASCII escaper from the mach compiler's mach.cli.json into std.data.json as the one JSON emission home, and unify it with the tree emitter's structural escaper into a single escape core. escape_unit takes an explicit policy: ESCAPE_VERBATIM (structural, UTF-8 bytes verbatim) drives the tree emitter emit(); ESCAPE_ENSURE_ASCII (RFC 3629 decode to \uXXXX, astral surrogate pairs, U+FFFD on invalid input) drives the streaming surface. The tree emitter now renders through a counting io.Writer (BufWriter) so both surfaces share one sink and one escaper; emit() keeps its measure-when-short semantics. Add the streaming NDJSON surface (Object, object_begin/object_end, field_str/field_str_or_null/field_i64/field_bool, write_json_string) ported verbatim from mach.cli.json, with its exact-byte tests plus a full-object call-sequence test. Tree and streaming output are byte-identical to their pre-unification form. The #337/#340 parse/emit representation asymmetry warnings are preserved. Closes #338
…ters Re-port the streaming NDJSON surface from the current mach dev, which mach#1838 extended past the flat-object surface the first port captured. Add the Array record and the nesting primitives object_end_value, field_object_begin, field_array_begin, array_end, array_object_begin, and field_null, ported verbatim with their Object.first / Array.first comma-state machine intact. object_end now closes via object_end_value and writes the sole NDJSON line terminator, so a whole nested object still lands on one line. Port mach#1838's exact-byte compose tests (nested object + array-of- objects, empty array) and exercise field_null in the flat sequence test. The full public surface and byte output match the compiler's current mach.cli.json, so the mach-side rewire is a use swap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #338.
What
Moves the streaming ASCII escaper out of the mach compiler's
mach.cli.jsonand intostd.data.jsonas the one JSON emission home, unifying it with the tree emitter's#337structural escaper into a single escape core.One escaper, explicit policy knob
Both emission surfaces now escape string bodies through one core (
escape_unit) selected byEscapePolicy:ESCAPE_VERBATIM(structural-utf8-verbatim) — escapes",\, and the C0 controls only; every other byte, including UTF-8 lead/continuation bytes, passes through verbatim. The tree emitteremit()uses this, so valid UTF-8 round-trips as UTF-8.ESCAPE_ENSURE_ASCII— as above for bytes< 0x80, but each byte>= 0x80is UTF-8 decoded (full RFC 3629 validation) and emitted as a\uXXXXescape (astral code points as a surrogate pair); invalid UTF-8 emits the U+FFFD replacement escape, so every emitted byte is printable ASCII. The streaming NDJSON surface uses this.The tree emitter now renders through a counting
io.Writer(BufWriter) instead of a bespokeEmitter, so both surfaces share one sink and one escaper.emit()keeps its exact measure-when-buffer-short semantics (returns the required length even when the buffer is too small).Streaming NDJSON surface (full flat + nested/array)
Ported verbatim from the current-dev
mach.cli.json(which mach#1838 extended past the original flat surface):Object,Array,object_begin/object_end, the nesting primitivesobject_end_value/field_object_begin/field_array_begin/array_end/array_object_begin, and the membersfield_str/field_str_or_null/field_null/field_i64/field_bool, pluswrite_json_string. TheObject.first/Array.firstcomma-state machine is ported unchanged.object_endcloses viaobject_end_valueand writes the sole NDJSON line terminator, so a whole nested object lands on one line. Public names match the compiler's exactly, so the mach-side rewire (refactor/338-json-to-std, rewiring bothtesting.machanddiagnostic.mach/flush_json) is auseswap with no call-site churn.Placement rationale
Kept in
std.data.jsonitself (not a sibling module). The one escape core stays private, shared by both public emission surfaces; a sibling module would force exporting the escaper or adding a third module just to share it. "One home" = one module.Byte-identity
emittests (null/number/bool, array/object round-trips, quote/backslash/control escapes, object keys) all still pass unchanged.write_json_stringtests were ported over frommach.cli.json(identical inputs andwantbyte arrays); mach#1838's compose tests (nested object + array-of-objects{"a":1,"loc":{"line":2,"col":3},"rel":[{"label":"x"},{"label":"y"}]}\n, and empty array{"rel":[]}\n) were ported verbatim; and a flat full-object sequence test asserts{"n":7,"s":"a\"b","b":true,"o":null,"z":null}\nbyte-for-byte (exercisingfield_null).Scope
#337/#340parse/emit representation-asymmetry warnings preserved intact;#340stays open.CHANGELOG.md[Unreleased]entry added.Verification
mach build .andmach test .green — 588 passed, 0 failed (36 instd.data.json: 23 tree + 13 streaming). Both CI checks (build + cross-riscv64) pass.🤖 Generated with Claude Code