You don't anticipate anything possibly going wrong in the transition, right?

During the time that the script is running, the sorting on some category pages gets weird -- newly-updated pages get the new sorting while older pages are still in the old sorting, and using "next page" doesn't work as you'd expected.

But those problems get corrected as the script reaches that category, and when the script is done, everything works the way that it's supposed to.

The script took about four hours to run when converted Swedish Wikipedia. I'd expect it to be shorter than that for Italian, because there are fewer pages.

English WP took six days, but that's the biggest by far, and it turns out the process is exponential rather than arithmetic. :)

Could it be written a proper page describing how this subsystem works and what the expected impact will be? It seems like the implemented system is somewhat different from the announced system.

Could it be written a proper page describing how this subsystem works and what the expected impact will be?

@jeblad: Documentation can be found at https://www.mediawiki.org/wiki/Manual:$wgCategoryCollation. In most of these cases (except Hebrew and Norwegian), the wiki is switching from uca-<langcode> to uca-<langcode>-u-kn, in which case the only difference is the addition of the numeric sorting feature. For Hebrew and Norwegian, they are starting from uppercase collation. Norwegian can either upgrade to uca-no-u-kn (which is Unicode Collation Algorithm tailored for Norwegian + numeric sorting) or numeric (which is identical to what they have now, but with numeric sorting). Hebrew isn't supported by our IcuCollation class, so they can only upgrade to numeric (unless someone modifies IcuCollation to support Hebrew in the very near future).

How will these three blocks be sorted

• foo 123 456 bar
• foo 456 123 bar
• foo 789 bar
• foo 789

• foo 123.456 bar
• foo 456.123 bar
• foo 789 bar
• foo 789

• foo 123,456 bar
• foo 456,123 bar
• foo 789 bar
• foo 789

@jeblad: Numeric sorting only works for unbroken sequences of digits. Digits separated by commas, periods, or spaces are treated as separate numbers (and thus may still require DEFAULTSORT keys).

Change 316486 had a related patch set uploaded (by Kaldari):
Switching 10 wikis to numeric category collation per T146675

https://gerrit.wikimedia.org/r/316486

Change 316486 merged by jenkins-bot:
Switching 10 wikis to numeric category collation per T146675

https://gerrit.wikimedia.org/r/316486

Mentioned in SAL (#wikimedia-operations) [2016-10-19T23:20:11Z] <dereckson@mira> Synchronized wmf-config/InitialiseSettings.php: Switching 10 more wikis to numeric category collation (T146675) (duration: 00m 59s)

I'll start running the updateCollation.php script for each wiki tonite.

URLs for testing...

bnwiki: https://bn.wikipedia.org/wiki/%E0%A6%AC%E0%A6%BF%E0%A6%B7%E0%A6%AF%E0%A6%BC%E0%A6%B6%E0%A7%8D%E0%A6%B0%E0%A7%87%E0%A6%A3%E0%A7%80:%E0%A6%85%E0%A6%B8%E0%A6%AE%E0%A7%8D%E0%A6%AA%E0%A7%82%E0%A6%B0%E0%A7%8D%E0%A6%A3

bnwikisource: Could not find any affected categories.

cswiki: https://cs.wikipedia.org/wiki/Kategorie:Americk%C3%A9_ak%C4%8Dn%C3%AD_filmy

frwiki: https://fr.wikipedia.org/wiki/Cat%C3%A9gorie:Com%C3%A9die_dramatique_fran%C3%A7aise

hewiki: https://he.wikipedia.org/wiki/%D7%A7%D7%98%D7%92%D7%95%D7%A8%D7%99%D7%94:%D7%A1%D7%93%D7%A8%D7%95%D7%AA_%D7%98%D7%9C%D7%95%D7%95%D7%99%D7%96%D7%99%D7%94_%D7%90%D7%9E%D7%A8%D7%99%D7%A7%D7%90%D7%99%D7%95%D7%AA_%D7%A9%D7%A2%D7%9C%D7%95_%D7%9C%D7%A9%D7%99%D7%93%D7%95%D7%A8_%D7%91%D7%A9%D7%A0%D7%95%D7%AA_%D7%94-2000

huwiki: https://hu.wikipedia.org/wiki/Kateg%C3%B3ria:A_Naprendszer_kisbolyg%C3%B3i

itwiki: https://it.wikipedia.org/wiki/Categoria:Serie_televisive_statunitensi

plwiki: https://pl.wikipedia.org/wiki/Kategoria:Dywizje_piechoty_III_Rzeszy

ruwiki: https://ru.wikipedia.org/wiki/%D0%9A%D0%B0%D1%82%D0%B5%D0%B3%D0%BE%D1%80%D0%B8%D1%8F:%D0%A2%D0%B5%D0%BB%D0%B5%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB%D1%8B_%D0%BF%D0%BE_%D0%B0%D0%BB%D1%84%D0%B0%D0%B2%D0%B8%D1%82%D1%83

viwiki: https://vi.wikipedia.org/wiki/Th%E1%BB%83_lo%E1%BA%A1i:Ch%C6%B0%C6%A1ng_tr%C3%ACnh_truy%E1%BB%81n_h%C3%ACnh_ng%C3%B4n_ng%E1%BB%AF_ti%E1%BA%BFng_Anh

Well, there is a problem. 99<019<101

@DannyH and @kaldari.

In T146675#2730863, @IKhitron wrote:

Well, there is a problem. 99<019<101

Where are you seeing this?

w:he:category:disambiguations, @Niharika.

Thanks @IKhitron! I've filed a bug for that: T148774. I think I know exactly how to fix this, so it shouldn't take long.

Thank you!

Btw, how do you sort

• abc 20 5
• abc 5 20
• abc 5 80

? Can I assume that it wil be sorted by first number and then by second?

@IKhitron: Yes. I'm not 100% sure there are no bugs with complicated sequences of numbers, but I just tested it locally and got the following sort order:

• Abc 5 3
• Abc 5 20
• Abc 5 80
• Abc 20 5

Good, thanks.

@IKhitron: The patch to fix leading zeros has been merged. It should get deployed to he.wiki next Thursday. Then we can rebuild the sortkeys on Thursday evening or Friday.

Thanks, and it will be deployed next Wednesday, @kaldari.

Bokmål Wikipedia would like UCA while we're at it.

All the languages besides Norwegian are finished.

Change 317652 had a related patch set uploaded (by Kaldari):
Switch Norwegian Wikipedia to uca-no-u-kn category collation

https://gerrit.wikimedia.org/r/317652

Change 317652 merged by jenkins-bot:
Switch Norwegian Wikipedia to uca-no-u-kn category collation

https://gerrit.wikimedia.org/r/317652

@kaldari, something new?

For Hebrew and Norwegian, they are starting from uppercase collation. Norwegian can either upgrade to uca-no-u-kn (which is Unicode Collation Algorithm tailored for Norwegian + numeric sorting) or numeric (which is identical to what they have now, but with numeric sorting). Hebrew isn't supported by our IcuCollation class, so they can only upgrade to numeric (unless someone modifies IcuCollation to support Hebrew in the very near future)

I added he to IcuCollation in https://gerrit.wikimedia.org/r/318674

@kaldari, or maybe @DannyH? When can we expect the rerun? Thank you.

@IKhitron: Rerunning now. Should be done in a few hours. Sorry for the delay.

Thanks!

@IKhitron: All fixed now: https://he.wikipedia.org/wiki/%D7%A7%D7%98%D7%92%D7%95%D7%A8%D7%99%D7%94:%D7%A4%D7%99%D7%A8%D7%95%D7%A9%D7%95%D7%A0%D7%99%D7%9D

Thank you very much!