GH-96172 fix unicodedata.east_asian_width being wrong on unassigned code points #96207
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
also return the correct width for unassigned but reserved characters according to EastAsianWidth.txt
- this guards against accidentally introducing changes in the future - if east_asian_width had been part of the checksum from the beginning, the bug would have been found much earlier by PyPy
isidentical
reviewed
Aug 24, 2022
|
@cfbolz do you know whether we handle the first case mentioned in "6.1 Unassigned and Private-Use Characters" (Wide for CJK): |
|
@isidentical oops, I definitely had an explicit test, seems that got lost somehow. will retrieve it tomorrow. and yes, it definitely had characters from the CJK ranges that you list |
|
it's in my first commit, but then in the third one I removed it again for some reason :-/. 5688e6a#diff-c31ff7b8fca97de6b4fdaca3e14da27ab3cac411653e9c510a5378b189f909eaR223 fixing tomorrow. |
|
Amazing, thanks! |
isidentical
approved these changes
Aug 25, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
unicodedata.east_asian_widthreturned the wrong value in two situations:This is fixed by changing the default return value through reordering the list
EASTASIANWIDTH_NAMESand by adding records for unassigned but reserved entries.