i18n: further work, expr support #8292

RenjiSann · 2025-06-30T20:40:06Z

Replaces #7986

Internationalization of `expr`

Handles non-UTF8 inputs without panicking
Support for locale-aware string comparison (which solves expr: 50n \> -51 is producing different result than GNU expr #5378)
Support for locale encoding aware for length, substr and index, which now support
- However, match can't be ported, as the regex engine we use (onig) doesn't handle non UTF-8 input (someday we will Replace Oniguruma #1145)
- This makes the GNU test expr-multibyte.pl pass at 75%, match tests are still missing. See expr is failing with multibyte chars #3132

Additions to `uucore::i18n`

Refactored the feature dependency tree to optimize compile times
Added a decimal module that allows one to access decimal and grouping separators etc...

github-actions · 2025-06-30T21:40:58Z

GNU testsuite comparison:

Congrats! The gnu test tests/fmt/goal-option is no longer failing!

github-actions · 2025-07-01T00:12:00Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)

src/uucore/src/lib/features/i18n/collator.rs

github-actions · 2025-07-01T10:59:11Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/stdbuf (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

sylvestre · 2025-07-01T12:01:03Z

src/uucore/src/lib/features/i18n/mod.rs

+        let mut split = locale_var_str.split(&['.', '@']);
+
+        if let Some(simple) = split.next() {
+            let bcp47 = simple.replace("_", "-");


maybe add a comment explaining what means bcp47 :)

github-actions · 2025-07-01T14:22:32Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-07-02T03:16:25Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/stdbuf (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

RenjiSann · 2025-07-04T00:54:37Z

src/uu/expr/src/expr.rs

+    #[error("Unsupported non-utf8 argument passed to match: {}", _0.quote())]
+    UnsupportedNonUtf8Match(String),


@sylvestre This is a uutils only error at the moment. I am not sure whether adding an error like this or calling expect() at the problematic code path would be better.
Also, because I consider this a bit like an internal error, I didn't bother adding a translation. I will do it if you deem it necessary.

RenjiSann · 2025-07-04T01:00:51Z

src/uu/expr/src/locale_aware.rs

+    let mut opts = CollatorOptions::default();
+    opts.alternate_handling = Some(AlternateHandling::Shifted); // This is black magic
+    let _ = try_init_collator(opts);


Here, I don't exactly understand how the CollatorOptions affect the collating. I've just toyed with it until #5378 passed. That's why I said it's black magic.

github-actions · 2025-07-04T02:16:05Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/stdbuf (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

cakebaker · 2025-07-06T12:04:50Z

There is a conflict :|

src/uucore/Cargo.toml

src/uu/expr/src/locale_aware.rs

github-actions · 2025-07-07T10:06:12Z

GNU testsuite comparison:

Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)
Congrats! The gnu test tests/tail/tail-c is no longer failing!

tests/by-util/test_expr.rs

github-actions · 2025-07-07T13:58:43Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-07-07T23:21:42Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

src/uucore/src/lib/features/i18n/mod.rs

cakebaker · 2025-07-08T09:28:45Z

src/uucore/src/lib/features/i18n/mod.rs

+            // locale. Treat the special case of the given locale being "C"
+            // which becomes the default locale.
+            let encoding =
+                if (locale != DEFAULT_LOCALE || bcp47 == "C") && split.next() == Some("UTF-8") {


I think split.next() == Some("UTF-8") should be case-insensitive. At least GNU expr doesn't seem to care about it:

$ LC_ALL=de_CH.UTF-8 expr length äöü 3 $ LC_ALL=de_CH.utf-8 expr length äöü 3

Interesting, I didn't think of that

- Add locale comparison test (issue uutils#5378) - Implement GNU expr-multibyte test

RenjiSann mentioned this pull request Jun 30, 2025

i18n: Collation draft #7986

Closed

RenjiSann force-pushed the collating branch 2 times, most recently from 26a0eee to 644ea84 Compare June 30, 2025 20:46

RenjiSann force-pushed the collating branch from 13bf589 to 92b4264 Compare June 30, 2025 23:17

RenjiSann force-pushed the collating branch from 92b4264 to 7b07a09 Compare July 1, 2025 09:43

sylvestre reviewed Jul 1, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/collator.rs Show resolved Hide resolved

RenjiSann force-pushed the collating branch from 7b07a09 to 4f82d48 Compare July 1, 2025 10:06

sylvestre reviewed Jul 1, 2025

View reviewed changes

RenjiSann force-pushed the collating branch from 4f82d48 to 7436bb2 Compare July 1, 2025 13:28

RenjiSann force-pushed the collating branch from 7436bb2 to ec5c28e Compare July 2, 2025 02:23

RenjiSann force-pushed the collating branch 3 times, most recently from 8ad5eb5 to 4b5b0d1 Compare July 4, 2025 00:48

RenjiSann requested a review from sylvestre July 4, 2025 00:48

RenjiSann commented Jul 4, 2025

View reviewed changes

RenjiSann force-pushed the collating branch 2 times, most recently from da3c388 to 9660754 Compare July 4, 2025 01:22

RenjiSann marked this pull request as ready for review July 4, 2025 03:13

cakebaker reviewed Jul 6, 2025

View reviewed changes

src/uucore/Cargo.toml Outdated Show resolved Hide resolved

cakebaker reviewed Jul 7, 2025

View reviewed changes

src/uu/expr/src/locale_aware.rs Show resolved Hide resolved

RenjiSann force-pushed the collating branch 2 times, most recently from bfb5cc6 to 7987ade Compare July 7, 2025 09:20

cakebaker reviewed Jul 7, 2025

View reviewed changes

tests/by-util/test_expr.rs Outdated Show resolved Hide resolved

cakebaker reviewed Jul 7, 2025

View reviewed changes

tests/by-util/test_expr.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the collating branch from 7987ade to b6a9595 Compare July 7, 2025 13:13

RenjiSann force-pushed the collating branch 2 times, most recently from 2b7d214 to c0538f0 Compare July 7, 2025 22:37

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/mod.rs Outdated Show resolved Hide resolved

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/mod.rs Outdated Show resolved Hide resolved

cakebaker reviewed Jul 8, 2025

View reviewed changes

src/uucore/src/lib/features/i18n/mod.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the collating branch from c0538f0 to c5e884d Compare July 8, 2025 09:19

cakebaker reviewed Jul 8, 2025

View reviewed changes

RenjiSann force-pushed the collating branch from c5e884d to d9357fa Compare July 8, 2025 09:39

RenjiSann added 8 commits July 8, 2025 12:02

i18n: small refactor, add decimal stuff

bb8744f

i18n: add collating

4b78537

expr: Handle non UTF8 inputs

bb6820a

expr: add locale-aware index

80040ec

expr: add locale-aware comparison

b31a9f0

expr: add locale-aware length

85ae4f6

expr: add locale-aware substr

c5c4b97

test(expr): add locale-related tests, remove deprecated tests

4034791

- Add locale comparison test (issue uutils#5378) - Implement GNU expr-multibyte test

RenjiSann force-pushed the collating branch from d9357fa to 4034791 Compare July 8, 2025 10:02

		#[error("Unsupported non-utf8 argument passed to match: {}", _0.quote())]
		UnsupportedNonUtf8Match(String),

Uh oh!

i18n: further work, expr support #8292

Are you sure you want to change the base?

i18n: further work, expr support #8292

Conversation

RenjiSann commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Internationalization of expr

Additions to uucore::i18n

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

sylvestre Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

RenjiSann Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

RenjiSann Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

RenjiSann Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 4, 2025

Uh oh!

cakebaker commented Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cakebaker Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

RenjiSann Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

RenjiSann Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RenjiSann commented Jun 30, 2025 •

edited

Loading

Internationalization of `expr`

Additions to `uucore::i18n`

RenjiSann Jul 4, 2025 •

edited

Loading