fix(compiler/tokenizer): track lines across multiline comments (#668)#669
Open
wilmveel wants to merge 2 commits into
Open
fix(compiler/tokenizer): track lines across multiline comments (#668)#669wilmveel wants to merge 2 commits into
wilmveel wants to merge 2 commits into
Conversation
Comment tokens (both single-line `//...\n` and multiline `/* ... */`) embed newlines in their value, but `nextCoordinates` only advanced the line counter for `NewLine` tokens. As a result every line spanned by a multiline comment was lost from the line count, so parser errors after such a comment pointed to an earlier, incorrect line. Count embedded newlines for non-`NewLine` tokens and advance the line counter accordingly, resetting the column to the position after the final newline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #668 — compile errors pointed to the wrong (earlier) line whenever a multiline comment preceded the offending token.
Comment tokens carry their full text as the token value. Single-line comments (
^//.*\n) include a trailing newline, and multiline comments (/* ... */) include every embedded newline. However,Coordinates.nextCoordinatesonly advanced thelinecounter forNewLinetokens — comments fell through to theelsebranch, which advances onlyposition/idx. Every line spanned by a multiline comment was therefore dropped from the line count, so any parser error reported after such a comment pointed to a line higher up in the file.Fix
In
Tokenizer.nextCoordinates, for non-NewLinetokens, count the newlines embedded in the token value. If there are any, advancelineby that count and resetpositionto the column after the final newline. Tokens without embedded newlines keep the existing fast path.Testing
ParserReferenceTest.shouldReportCorrectLineAfterMultiLineComment, which places an unresolvable reference after a 5-line block comment and asserts the error is reported on the correct line../gradlew :src:compiler:core:jvmTestpasses (existing coordinate assertions unaffected —idxAndLengthaccumulation is unchanged).🤖 Generated with Claude Code