Skip to content

Lexer location fixes and tests - #74

Merged
wtholliday merged 2 commits into
mainfrom
lexer-loc-test
Oct 14, 2026
Merged

wtholliday merged 2 commits into
mainfrom
lexer-loc-test

Conversation

@wtholliday

@wtholliday wtholliday commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #73.

  • Count raw newlines inside string literals. The lexer only advanced the line counter for whitespace and comment newlines, so every token after a multi-line string reported the wrong line and a column measured from an earlier line. Raw newlines in a string now count, and the string token keeps the line of its opening quote.
  • Replace the tok == Endl check with a pending_lines counter. Both the newline handler and the string scanner bump it, and the line advances when the next token is lexed. This removes the dependency on tok always holding the previously returned token. The parser's struct-literal backtrack saves and restores the counter with the rest of the lexer state.
  • The doc comment on Lexer::loc claimed the column was computed on demand. It is set eagerly for every token.
  • Add test_token_locations, pinning the Endl position at the newline on the line it ends, the deferred line advance across blank lines, the newline after a // comment, locations after a multi-line string, and where End lands after a trailing newline.

Before, for a stray } on line 5 after a two-line string:

str.lyte:4:1: Expected expression

After:

str.lyte:5:1: Expected expression

Test plan

  • cargo test --workspace passes (unit tests plus all four golden runners)
  • rustfmt --check clean on the touched files

🤖 Generated with Claude Code

wtholliday and others added 2 commits October 13, 2026 22:40
The comment on Lexer::loc said the column was computed on demand, but
it has been set eagerly for every token. Add a unit test pinning the
Endl location and the deferred line advance introduced in #73.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The lexer only advanced the line counter for whitespace and comment
newlines, so every token after a multi-line string literal reported
the wrong line and a column measured from an earlier line.

Replace the "previous token was Endl" check with a pending_lines
counter that both the newline handler and the string scanner bump.
The line advances when the next token is lexed, so the string token
keeps the line of its opening quote. The parser's struct-literal
backtrack now saves and restores the counter along with the rest of
the lexer state.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@wtholliday wtholliday changed the title Test token locations and fix stale lexer doc comment Lexer location fixes and tests Oct 14, 2026
@wtholliday
wtholliday merged commit bbc2f48 into main Oct 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant