Repository navigation
fix(65076): treat literal U+FFFD as a real rune in regex classes - #65080
Open
Timur Vural (timurzvural) wants to merge 1 commit into
Open
Timur Vural (timurzvural) wants to merge 1 commit into
Timur Vural (timurzvural) wants to merge 1 commit into
Conversation
utf8.RuneError is U+FFFD; only size==1 means invalid UTF-8. The old check treated a real replacement character as a decode error and fed 0xEF into character-class range checks. Co-authored-by: Cursor <cursoragent@cursor.com>
typescript-automation
Bot
added
the
For Uncommitted Bug
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the regex scanner’s U+FFFD handling fix.
No open issues 🎉
What changed in this PR
Fixes #65076 by distinguishing literal U+FFFD characters from invalid UTF-8 in the native compiler’s regex scanner, preventing false character-class range diagnostics.
Changes:
- Treats
utf8.RuneErroras invalid UTF-8 only when decoding consumes one byte. - Adds scanner regression tests for literal and escaped U+FFFD, including range boundaries and the
uflag.
| File | Description |
|---|---|
tsc/internal/scanner/scanner_test.go |
Adds five regression cases covering U+FFFD handling in regex literals. |
tsc/internal/scanner/regexp.go |
Corrects UTF-8 decoding checks and handles empty input separately. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #65076.
Bug
utf8.RuneErroris defined as U+FFFD.utf8.DecodeRunereturns that value for both a real U+FFFD (size 3) and an invalid UTF-8 byte (size 1).scanSourceCharactertreated anych == utf8.RuneErroras a decode failure and consumed a single raw byte (0xEF, the lead byte of UTF-8 U+FFFD).A class like
/[\uFFF9-�]/was therefore checked asU+FFF9 … 0xEF, producing falseTS1517(andTS1516with theuflag). Escaped\uFFFDwas fine; TypeScript 6.0.3 was fine.Fix
Only treat
ch == utf8.RuneError && size == 1as invalid UTF-8 (same pattern already used in the printer). A real U+FFFD advances by 3 bytes and participates in the range normally.Tests
TestRegExpLiteralReplacementCharacterInClassRangecovers the #65076 repro, theu-flag variant, escaped\uFFFD, a lone literal U+FFFD, and a range that starts with literal U+FFFD.AI assistance disclosure
This PR was authored with assistance from Cursor Agent. I have read and understand the contents of this PR.