[Feature #19908] Update to Unicode 15.1.0 #12798

ima1zumi · 2025-02-24T03:14:27Z

https://bugs.ruby-lang.org/issues/19908

Summary

Unicode 15.1.0 introduced a new grapheme cluster rule to handle Indic conjuncts. In UAX #29, rule GB9c was added to prevent grapheme breaks within certain Indic consonant+virama sequences. As a result, an orthographic syllable in scripts like Devanagari (e.g. “क्या”, consisting of KA + VIRAMA + YA) is now treated as a single extended grapheme cluster rather than split into two. This aligns the default segmentation with Indic writing system expectations.

UAX #29: Unicode Text Segmentation

New Enumerated Property

To support this, the Unicode Character Database (UCD) added a new enumerated property, Indic_Conjunct_Break (InCB), with values Consonant, Linker, Extend, and None, derived from existing properties. For example, virama characters in certain Brahmic scripts are classified as InCB=Linker, and base consonants as InCB=Consonant. These property values are listed in the UCD data (e.g. in DerivedCoreProperties.txt).

Impact on Ruby

This change affected Ruby’s implementation of Unicode support in several ways:

enc-unicode.rb
- Ruby’s enc-unicode.rb script, which processes Unicode data, had to be updated to handle the new enumerated property. Previously, it expected only binary properties.
Regex Engine
- Support for InCB property values was added to Ruby’s regex engine (exposing them as InCB_Linker, InCB_Consonant, etc.).
Grapheme Cluster Logic
- Ruby’s grapheme cluster regex (\X) logic was updated to incorporate GB9c, ensuring that Indic conjuncts are not split in string operations.

kou · 2025-02-24T03:23:01Z

NEWS.md

@@ -31,7 +36,6 @@ The following bundled gems are promoted from default gems.
 * reline 0.6.0
 * readline 0.0.4
 * fiddle 1.1.6
-


Could you revert a needless change?

I have reverted.

kou · 2025-02-24T03:27:59Z

tool/enc-unicode.rb

+      elsif /^(\h+)(?:\.\.(\h+))?\s*;\s*(\w(?>[\w\s;]*\w)?)/ =~ line
        $2 ? cps.concat(($1.to_i(16)..$2.to_i(16)).to_a) : cps.push($1.to_i(16))
+        current = $3.gsub(/\W+/, '_')


Could you add a comment that shows an example matched line and $1/$2/$3 for the case for easy to understand?

I have added some comments. https://github.com/ruby/ruby/pull/12798/files#diff-66b721e4f3da90f52f789b006bac8743157852b8bea2643ce866d70296acaec9R171-R179

ima1zumi · 2025-02-24T06:37:48Z

regparse.c

+        /* conjunctCluster := \p{InCB=Consonant} ([\p{InCB=Extend} \p{InCB=Linker}]* \p{InCB=Linker} [\p{InCB=Extend} \p{InCB=Linker}]* \p{InCB=Consonant})+ */
+        {
+          Node **CC_list = core_alts + 6; /* size: 2 */
+          R_ERR(create_property_node(CC_list+0, env, "InCB_Consonant"));
+
+          {
+            Node **CC_list1 = CC_list + 1; /* size: 4 */
+            // [\p{InCB=Extend} \p{InCB=Linker}]*
+            {
+              {
+                Node **CC_alt1 = CC_list1 + 1;
+                R_ERR(create_property_node(CC_alt1+1, env, "InCB_Extend"));
+                R_ERR(create_property_node(CC_alt1+2, env, "InCB_Linker"));
+
+                R_ERR(create_node_from_array(ALT, CC_alt1, CC_alt1+1));
+              }
+
+              R_ERR(quantify_node(CC_list1+1, 0, REPEAT_INFINITE));
+            }
+
+            // \p{InCB=Linker}
+            R_ERR(create_property_node(CC_list1+2, env, "InCB_Linker"));
+
+            // [\p{InCB=Extend} \p{InCB=Linker}]*
+            {
+              {
+                Node **CC_alt2 = CC_list1 + 3;
+                R_ERR(create_property_node(CC_alt2+1, env, "InCB_Extend"));
+                R_ERR(create_property_node(CC_alt2+2, env, "InCB_Linker"));
+
+                R_ERR(create_node_from_array(ALT, CC_alt2, CC_alt2+1));
+              }
+
+              R_ERR(quantify_node(CC_list1+3, 0, REPEAT_INFINITE));
+            }
+
+            // \p{InCB=Consonant}
+            R_ERR(create_property_node(CC_list1+4, env, "InCB_Consonant"));
+
+            R_ERR(create_node_from_array(LIST, CC_list1, CC_list1+1));
+            R_ERR(quantify_node(CC_list1, 1, REPEAT_INFINITE));
+          }
+          R_ERR(create_node_from_array(LIST, core_alts+5, CC_list));
+        }
+
        /* [^Control CR LF] */
-        core_alts[5] = node_new_cclass();
-        if (IS_NULL(core_alts[5])) goto err;
-        cc = NCCLASS(core_alts[5]);
+        core_alts[6] = node_new_cclass();
+        if (IS_NULL(core_alts[6])) goto err;
+        cc = NCCLASS(core_alts[6]);


This part is covered by an automatically generated test in this file based on GraphemeBreakTest.txt.

https://github.com/ruby/ruby/blob/dfc25204235079e23eadf9e0ba860c1ebcb14325/test/ruby/enc/test_grapheme_breaks.rb
https://www.unicode.org/Public/15.1.0/ucd/auxiliary/GraphemeBreakTest.txt

regparse.c

tool/enc-unicode.rb

…nCB)=Linker. https://www.unicode.org/reports/tr29/tr29-43.html#GB9c

…failure with Ruby 3.5.0dev This commit addresses the following failure with Ruby 3.5.0dev since ruby/ruby@6670926 ```ruby % ruby -v ruby 3.5.0dev (2025-03-21T06:17:15Z master d868922ea8) +PRISM [arm64-darwin24] % bin/test test/multibyte_chars_test.rb -n test_should_compute_grapheme_length Running 90 tests in parallel using 10 processes Run options: -n test_should_compute_grapheme_length --seed 52859 F Failure: MultibyteCharsExtrasTest#test_should_compute_grapheme_length [test/multibyte_chars_test.rb:512]: "त्र". Expected: 2 Actual: 1 bin/test test/multibyte_chars_test.rb:595 Finished in 0.209643s, 4.7700 runs/s, 38.1601 assertions/s. 1 runs, 8 assertions, 1 failures, 0 errors, 0 skips % ``` According to ruby/ruby#12798 ,this is an expected change since Unicode 15.1.0. > As a result, an orthographic syllable in scripts like Devanagari (e.g. “क्या”, consisting of KA + VIRAMA + YA) > is now treated as a single extended grapheme cluster rather than split into two. Fix rails#54794

ima1zumi force-pushed the unicode-15.1.0 branch from 1b0a49a to b3ec885 Compare February 24, 2025 03:20

ima1zumi changed the title ~~Update to Unicode 15.1.0~~ [Feature #19908] Update to Unicode 15.1.0 Feb 24, 2025

kou reviewed Feb 24, 2025

View reviewed changes

ima1zumi force-pushed the unicode-15.1.0 branch from b3ec885 to 162d060 Compare February 24, 2025 05:09

ima1zumi commented Feb 24, 2025

View reviewed changes

ima1zumi force-pushed the unicode-15.1.0 branch from 162d060 to 4817ab0 Compare February 24, 2025 14:38

ima1zumi marked this pull request as ready for review February 24, 2025 15:05

ima1zumi marked this pull request as draft February 25, 2025 00:21

tompng reviewed Feb 26, 2025

View reviewed changes

regparse.c Outdated Show resolved Hide resolved

ima1zumi force-pushed the unicode-15.1.0 branch 2 times, most recently from bcd7641 to d07beeb Compare March 4, 2025 11:16

ima1zumi marked this pull request as ready for review March 4, 2025 11:16

This comment has been minimized.

Sign in to view

tompng reviewed Mar 7, 2025

View reviewed changes

regparse.c Outdated Show resolved Hide resolved

ima1zumi force-pushed the unicode-15.1.0 branch 2 times, most recently from 40d6ec9 to c648189 Compare March 9, 2025 11:48

tompng reviewed Mar 9, 2025

View reviewed changes

tool/enc-unicode.rb Outdated Show resolved Hide resolved

ima1zumi force-pushed the unicode-15.1.0 branch from c648189 to 49775b7 Compare March 9, 2025 15:54

nurse self-assigned this Mar 13, 2025

nurse approved these changes Mar 14, 2025

View reviewed changes

ima1zumi added 3 commits March 18, 2025 20:30

Support for Indic_Conjunct_Break in Unicode 15.1

b006afb

[Feature #19908] Update Unicode headers to 15.1.0

9f0b398

Do not break within certain combinations with Indic_Conjunct_Break (I…

a66a4d3

…nCB)=Linker. https://www.unicode.org/reports/tr29/tr29-43.html#GB9c

ima1zumi force-pushed the unicode-15.1.0 branch from 49775b7 to a66a4d3 Compare March 18, 2025 11:30

ima1zumi merged commit 6670926 into ruby:master Mar 18, 2025
80 checks passed

yahonda mentioned this pull request Mar 23, 2025

Address MultibyteCharsExtrasTest#test_should_compute_grapheme_lengthfailure with Ruby 3.5.0dev rails/rails#54802

Merged

4 tasks

ima1zumi deleted the unicode-15.1.0 branch April 5, 2025 16:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature #19908] Update to Unicode 15.1.0 #12798

[Feature #19908] Update to Unicode 15.1.0 #12798

ima1zumi commented Feb 24, 2025 •

edited

Loading

kou Feb 24, 2025

ima1zumi Feb 24, 2025

kou Feb 24, 2025

ima1zumi Feb 24, 2025

ima1zumi Feb 24, 2025

This comment has been minimized.

[Feature #19908] Update to Unicode 15.1.0 #12798

[Feature #19908] Update to Unicode 15.1.0 #12798

Conversation

ima1zumi commented Feb 24, 2025 • edited Loading

Summary

New Enumerated Property

Impact on Ruby

kou Feb 24, 2025

Choose a reason for hiding this comment

ima1zumi Feb 24, 2025

Choose a reason for hiding this comment

kou Feb 24, 2025

Choose a reason for hiding this comment

ima1zumi Feb 24, 2025

Choose a reason for hiding this comment

ima1zumi Feb 24, 2025

Choose a reason for hiding this comment

This comment has been minimized.

ima1zumi commented Feb 24, 2025 •

edited

Loading