Allow U+2028 and U+2029 as line terminator for single-line comments?

[chirsz-ever]

ECMA-262 treat U+2028 and U+2029 as line terminators. They can be used to end line comments in JavaScript.

See JSONC-org/JSONC#23

[bradfitz]

Oh gross. TIL.

I'd kinda rather not. I'd almost rather detect them and fail the whole parse as being ambiguous. :)

[creachadair]

I agree with @bradfitz we probably shouldn't.

HuJSON/JWCC is based on the JSON grammar, rather than JavaScript, and JSON has a narrow definition of whitespace (including only space, newline, carriage return, and horizontal tab).

[dsnet]

Ugh, what a mess. To add complexity to the issue. U+2028 and U+2029 is valid in a JSON string literal according to RFC 8259, section 12. If you comment out a line with a JSON string literal with such characters, then the behavior would now be dependent on whether we respect such characters as terminating the comment or not.

I agree with @bradfitz to just reject the parse. Security vulnerabilities often arise by abusing ambiguous semantic meanings.

[twitchyliquid64]

Yeah, this is uniquely terribubble. A JSON parser that recognizes these characters will miss a key/val declaration that a JSON parser which does not will have (or the other way around, brain hurty).

Agree with throwing an error if these characters appear outside of a string literal, and loudly yelling. If JSON was used for headers in a request that traversed multiple stacks, you could probably use this as a request smuggling/desync primitive.

[creachadair]

There is a pretty decent test suite from https://github.com/nst/JSONTestSuite, which I've used for testing my own implementation. We might want to consider using that here too.

Because, as noted, the spec is ambiguous on several points, it provides cases that must be accepted, must be rejected, or are implementation-dependent. (The latter are still interesting to make sure the parser doesn't crash).

[chirsz-ever]

For visibility, <LS> and <PS> will be used below to refer to U+2028 and U+2029.

Let's discuss the possible locations of <LS> and <PS> in different cases:

(A) In string literals:

{
    "key<LS>": "a <LS> string"
}

This is supported by JSON and ES2019.

(B) As whitespace characters

{
    "a": 1, <LS>
    "b": 2
}

This is rejected by JSON, and we should probably reject it as well.

(C) In a block comment

{
    /* this is a block comment <LS> */
    "a": 1,
    "b": 2
}

This would not cause any problems.

(D) In the middle of a line comment

{
    // "eo": "Saluton, <LS>mondo!",
    "en": "Hello, world!"
}

This should definitely be rejected; the example above shows that an accidental comment can cause a compilation error in JavaScript.

(E) At the end of a line comment

{
    // a line comment<LS>
    "a": 1
}

With some simple processing, cases like above can be accepted ¹, increasing compatibility with JavaScript. But in real-world scenarios, this situation should not occur at all. Theoretically, rejecting this case also helps maintain semantic consistency with JavaScript (single-line comments and the following line terminator are considered two separate tokens), which is logically consistent with (B) and (D).

Thus, accepting (A)(C) and rejecting (B)(D)(E), the corresponding grammar rules can be written as:

whitespaces          ::=  ' ' | '\t' | '\r' | '\n' | comment ;
comment              ::=  single_line_comment | block_comment ;
single_line_comment  ::=  "//" not_line_terminator* ;
not_line_terminator  ::=  [^\r\n\u2028\u2029] ;
block_comment        ::=  "/*" .*? "*/" ;

Footnotes

1. https://github.com/JSONC-org/JSONC/blob/a7424f219d71df5abc583650c3bd3e100156121f/grammar/jsonc.abnf#L29 ↩