Skip to content

[String] Unexpected codepoints in tests #38957

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
derrabus opened this issue Nov 1, 2020 · 7 comments
Closed

[String] Unexpected codepoints in tests #38957

derrabus opened this issue Nov 1, 2020 · 7 comments
Labels
Bug Help wanted Issues and PRs which are looking for volunteers to complete them. Keep open Status: Needs Review String

Comments

@derrabus
Copy link
Member

derrabus commented Nov 1, 2020 •
edited
Loading

Symfony version(s) affected: 5.1-dev

Description
When running test String test suite on macOS, the tests fail because they receive more codepoints than expected.

How to reproduce
Run the String test suite on macOS.

Additional context
Discovered while working on #38620

https://github.com/symfony/symfony/pull/38956/checks?check_run_id=1339604533#step:8:1216

There were 3 failures:

1) Symfony\Component\String\Tests\UnicodeStringTest::testCodePointsAt with data set #8 (array(2360, 2381), 'नमस्ते', 2)
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
 Array &0 (
     0 => 2360
     1 => 2381
+    2 => 2340
+    3 => 2375
 )

/Users/runner/work/symfony/symfony/src/Symfony/Component/String/Tests/AbstractUnicodeTestCase.php:56

2) Symfony\Component\String\Tests\UnicodeStringTest::testBytesAt with data set #8 (array(224, 164, 184, 224, 165, 141), 'नमस्ते', 2)
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
     3 => 224
     4 => 165
     5 => 141
+    6 => 224
+    7 => 164
+    8 => 164
+    9 => 224
+    10 => 165
+    11 => 135
 )

/Users/runner/work/symfony/symfony/src/Symfony/Component/String/Tests/AbstractAsciiTestCase.php:45

3) Symfony\Component\String\Tests\UnicodeStringTest::testLength with data set #9 (5, 'अनुच्छेद')
Failed asserting that 4 is identical to 5.

/Users/runner/work/symfony/symfony/src/Symfony/Component/String/Tests/AbstractAsciiTestCase.php:165
@derrabus derrabus added this to the 5.1 milestone Nov 1, 2020
@derrabus derrabus removed this from the 5.1 milestone Nov 1, 2020
@nicolas-grekas
Copy link
Member

I have the same errors on my Ubuntu and this is due to a broken ICU release that both OSes ship.

Take this one:

  1. Symfony\Component\String\Tests\UnicodeStringTest::testLength with data set Hidden field labels are displayed #9 (5, 'अनुच्छेद')
    Failed asserting that 4 is identical to 5.

Use the text selection in the browser to count the number of chars in that string, you'll find 5, not 4.

I didn't investigate which version of what precisely should be skipped, but there is nothing wrong with the code on our side.

@derrabus
Copy link
Member Author

derrabus commented Nov 3, 2020

Homebrew currently ships ICU4C 67.1. This is pretty much up to date. I tried to bump to 68.1, but… https://bugs.php.net/80310

@derrabus
Copy link
Member Author

derrabus commented Nov 3, 2020 •
edited
Loading

I could manage to compile php against icu4c 68.1 now, which is the most recent release. I can still reproduce the problem. Do you have any insights? Is this a defect of ICU4C or a problem of PHP not calling ICU correctly? I feel like we should report this to someone. 🤔

@nicolas-grekas
Copy link
Member

I don't know if this if for PHP or ICU. Maybe report to both to have a quicker response?

@xelan
Copy link
Contributor

xelan commented Dec 3, 2020

Hope I can help a bit with the follwing info:

As far as I know, macOS uses a different Unicode normalization than "the rest of the world". While Windows and Linux (and probably others) use the Normalization Form Canonical Composition, macOS uses Normalization Form Canonical Decomposition.

See also https://gist.github.com/JamesChevalier/8448512

Maybe a workaround would be to compare the normalized Unicode values using the PHP Normalizer, skipping the test if running on macOS and it is not available.

@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@derrabus derrabus added Help wanted Issues and PRs which are looking for volunteers to complete them. Keep open and removed Stalled labels Jun 4, 2021
@derrabus
Copy link
Member Author

derrabus commented Jun 5, 2021

The issue also pops up in our PHP 8.1 run on GitHub Actions (Ubuntu 20.04): https://github.com/symfony/symfony/runs/2752189826#step:6:6107

@derrabus derrabus changed the title [String] Unexpected codepoints in tests on macOS [String] Unexpected codepoints in tests Jun 6, 2021
@derrabus derrabus mentioned this issue Jun 6, 2021
Closed
derrabus referenced this issue Jun 28, 2021
@nicolas-grekas
Fix CI
1bbacd5
@nicolas-grekas nicolas-grekas mentioned this issue Jun 28, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Help wanted Issues and PRs which are looking for volunteers to complete them. Keep open Status: Needs Review String
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nicolas-grekas @derrabus @xelan @carsonbot