Edition 2024 reserved tokens are accepted

[ehuss]

The 2024 edition reserved unprefixed guarded string literals and two or more # characters. However, proc-macro2 is parsing these with the previous edition behavior. I'm not sure what the edition policy of proc-macro2 is, but I would somewhat expect these to be rejected.

use std::str::FromStr;

fn main() {
    for input in [
        "#\"foo\"#",
        "##",
    ] {
        println!("{:?}", proc_macro2::TokenStream::from_str(input));
    }
}

outputs:

Ok(TokenStream [Punct { char: '#', spacing: Alone }, Literal { lit: "foo" }, Punct { char: '#', spacing: Alone }])
Ok(TokenStream [Punct { char: '#', spacing: Joint }, Punct { char: '#', spacing: Alone }])

The edition chapter on the reservation: https://doc.rust-lang.org/nightly/edition-guide/rust-2024/reserved-syntax.html. This is from RFC 3593. The grammar from the reference is:

RESERVED_GUARDED_STRING_LITERAL -> `#`+ STRING_LITERAL

RESERVED_POUNDS -> `#`{2..}

Using proc-macro2 1.0.106.