PHP code highlight is broken when opening tag <? inside of string

[belogubovm]

Type: Bug

Issue is found in this extension using VS Code bisect.

Try this code snippet and you will see highlight in PHP files (and sometimes tokenization and other features) is broken.

function foo()
{
    $bar = '<?';

    return true;
}

foo('123');

Extension version: 1.3.3
VS Code version: Code 1.84.2 (1a5daa3a0231a0fbba4f14db7ec463cf99d7768e, 2023-11-09T10:50:47.800Z)
OS version: Linux x64 6.5.0-10-generic
Modes:

System Info

Item	Value
CPUs	AMD Ryzen 7 4700U with Radeon Graphics (8 x 2615)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	9, 6, 4
Memory (System)	15.04GB (9.46GB free)
Process Argv	--unity-launch --crash-reporter-id b9b97954-ebd4-48ff-8fed-a1c0b27c3b62
Screen Reader	no
VM	0%
DESKTOP_SESSION	plasma
XDG_CURRENT_DESKTOP	KDE
XDG_SESSION_DESKTOP	KDE
XDG_SESSION_TYPE	x11

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscoreces:30445986
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263:30880072
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
2e4cg342:30602488
f6dab269:30613381
vscrp:30673768
showlangstatbar:30737416
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30866567
pythonnoceb:30805159
synctok:30869157
dsvsc013:30795093
dsvsc014:30804076
dsvsc015:30845448
pythontestfixt:30871694
pythonregdiag2:30871582
pyreplss2:30886141
pythonmypyd1:30879173
pythoncet0:30885854
2e7ec940:30885897
pythontbext0:30879054
dsvsc016:30886110
dsvsc017cf:30886113
dsvsc018:30886114
aa_t_chat:30882232