Use rustc_lexer as fallback lexer

[CAD97]

Migrates the fallback parser to use rustc_lexer. WIP proof of concept currently.

Closes #201, closes #126, impacts #55, #5.

Failing tests:

• fail (parses ' static as Literal { lit: ' static, span: bytes(1..9) })
• no_panic (parses successfully)
• roundtrip (fails to terminate)
• span_test (lex error)
• test_debug_tokenstream (lex error)

[alexcrichton]

FWIW one of the goals of this crate is to compile as fast as possible, so mind taking some measurements of that for both before/after this change? (build time afterwards including that of rustc_lexer

[CAD97]

Very unscientific test:

cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ git.exe checkout master
Switched to branch 'master'
Your branch is up to date with 'upstream/master'.
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ cargo.exe clean
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ time cargo.exe build --release
   Compiling proc-macro2 v1.0.5 (D:\usr\Documents\Code\Rust\proc-macro2)
   Compiling unicode-xid v0.2.0
    Finished release [optimized] target(s) in 6.57s

real    0m6.678s
user    0m0.000s
sys     0m0.000s
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ git.exe checkout rustc_lexer
Switched to branch 'rustc_lexer'
Your branch is up to date with 'origin/rustc_lexer'.
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ time cargo.exe build --release
   Compiling rustc_lexer v0.1.0
   Compiling proc-macro2 v1.0.5 (D:\usr\Documents\Code\Rust\proc-macro2)
    Finished release [optimized] target(s) in 2.96s

real    0m3.036s
user    0m0.000s
sys     0m0.000s

So it looks like the removal of strnom is a notable benefit.

[CAD97]

Whoops, just realized I missed a cargo clean there, so here it is with a clean:

cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ cargo.exe clean
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ time cargo.exe build --release
   Compiling unicode-xid v0.2.0
   Compiling proc-macro2 v1.0.5 (D:\usr\Documents\Code\Rust\proc-macro2)
   Compiling rustc_lexer v0.1.0
    Finished release [optimized] target(s) in 3.53s

real    0m3.598s
user    0m0.000s
sys     0m0.016s

(measurements done with beta-x86_64-pc-windows-gnu / rustc 1.39.0-beta.5 (fa5c2f3e5 2019-10-02))

[CAD97]

Second test (with procmacro2_semver_exempt) shows an increase, so it's probably all within noise without controlling better.

cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ git checkout master
Switched to branch 'master'
Your branch is up to date with 'upstream/master'.
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ cargo.exe clean
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ RUSTFLAGS="--cfg procmacro2_semver_exempt" time cargo.exe build --release
   Compiling proc-macro2 v1.0.5 (D:\usr\Documents\Code\Rust\proc-macro2)
   Compiling unicode-xid v0.2.0
    Finished release [optimized] target(s) in 4.32s
0.00user 0.00system 0:04.39elapsed 0%CPU (0avgtext+0avgdata 232maxresident)k
0inputs+0outputs (0major+72minor)pagefaults 0swaps
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ git checkout rustc_lexer
Switched to branch 'rustc_lexer'
Your branch is up to date with 'origin/rustc_lexer'.
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ cargo.exe clean
cad97@DESKTOP-HIBC3H1:/mnt/d/usr/Documents/Code/Rust/proc-macro2$ RUSTFLAGS="--cfg procmacro2_semver_exempt" time cargo.exe build --release
    Updating crates.io index
   Compiling proc-macro2 v1.0.5 (D:\usr\Documents\Code\Rust\proc-macro2)
   Compiling unicode-xid v0.2.0
   Compiling rustc_lexer v0.1.0
    Finished release [optimized] target(s) in 5.19s
0.00user 0.00system 0:05.29elapsed 0%CPU (0avgtext+0avgdata 236maxresident)k
0inputs+0outputs (0major+72minor)pagefaults 0swaps

[dtolnay]

I think I would prefer to stick with this crate's own lexer for now.

Thanks anyway for the PR!