I'm running Ubuntu 19.04 on a ZFS mirrored pool, where my home partition is configured with 'utf8only=on' attribute. I've cloned cpython and after running the tests, as described in devguide.python.org, I have 11 test failures:

== Tests result: FAILURE ==

389 tests OK.

11 tests failed:
test_cmd_line_script test_httpservers test_imp test_import
test_ntpath test_os test_posixpath test_socket test_unicode_file
test_unicode_file_functions test_zipimport

I've been looking for similar or matching reported issues, but could not find one. I'm on the EuroPython 2019 CPython sprint and we'll be looking into this with the help of some of the core devs.

Here's some additional information I found for that specific attribute:

From the documentation at
http://dlc.sun.com/osol/docs/content/ZFSADMIN/gazss.html
(link is dead, but here's where I found the section below: https://zfs-discuss.opensolaris.narkive.com/3NqQVG0H/utf8only-and-normalization-properties#post1)

utf8only
Boolean
Off
This property indicates whether a file system should reject file names
that include characters that are not present in the UTF-8 character code
set. If this property is explicitly set to off, the normalization
property must either not be explicitly set or be set to none. The
default value for the utf8only property is off. This property cannot be
changed after the file system is created.

I think Dimiter was able to fix most of the failures, except test_unicode_file_functions.
Yesterday during the sprints we were looking at it, and we did some tests using the following snippet:

import os
import unicodedata
upsilon_diaeresis_and_hook = "ϔ"

for form in ["NFC", "NFD", "NFKC", "NFKD"]:                       
  unicode_filename = unicodedata.normalize(form, upsilon_diaeresis_and_hook)
  with open(unicode_filename, "w") as f: f.write(form)
  print("N:", ascii(unicode_filename))
  print([ascii(filename) for filename in os.listdir('.')])

On ext4 this creates 4 different files: ['\u03d4', '\u03d2\u0308', '\u03ab', '\u03a5\u0308']
On ZFS with utf8only=true (and I believe normalization=formD), only 2 files are created but each of the 4 filenames can be used to access either of the 2 files.
This is also the default behavior on Mac.

The test is already skipped on darwin (Lib/test/test_unicode_file_functions.py:120), and should be skipped for ZFS too (might depend on the exact flags used), however we weren't able to find a portable way to determine the filesystem and flags.

An alternative is to try creating the 4 files and skip the test if only 2 gets created and if all the names can be used to open these two files, however this might mask other failures. Unless someone can come up with a better way to do this, I think this is the only option.

In addition, different filesystems that don't exhibit this behavior can be used on Mac, so the test shouldn't be skipped in those cases.

"""
On ext4 this creates 4 different files: ['\u03d4', '\u03d2\u0308', '\u03ab', '\u03a5\u0308']
On ZFS with utf8only=true (and I believe normalization=formD), only 2 files are created but each of the 4 filenames can be used to access either of the 2 files.
This is also the default behavior on Mac.

The test is already skipped on darwin (Lib/test/test_unicode_file_functions.py:120), and should be skipped for ZFS too (might depend on the exact flags used), however we weren't able to find a portable way to determine the filesystem and flags.
"""

I suggest to create a temporary directory, create the 4 files and see how many files you can using os.listdir(). If you get 4, the FS doesn't normalize anything. If you get less, it's likely that the FS normalizes names.

Confirmed.

Repro: Do an ubuntu 20.04 install and choose "experimental zfs" support during install - https://ubuntu.com/blog/zfs-focus-on-ubuntu-20-04-lts-whats-new). On such a zfs filesystem, the following tests from a ./python -m test.regrtest run fail in 3.10:

11 tests failed:
test_cmd_line_script test_httpservers test_imp test_import
test_ntpath test_os test_posixpath test_socket test_unicode_file
test_unicode_file_functions test_zipimport

Move over to a tmpfs and all but test_httpservers now pass. test_httpservers tries to create such a path on /tmp

======================================================================
ERROR: test_undecodable_filename (test.test_httpservers.SimpleHTTPServerTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/greg/test/cpython/Lib/test/test_httpservers.py", line 400, in test_undecodable_filename
    with open(os.path.join(self.tempdir, filename), 'wb') as f:
OSError: [Errno 84] Invalid or incomplete multibyte or wide character: '/tmp/tmpnt9ch98x/@test_124227_tmp\udce7w\udcf0.txt'

I expect any filesystem mounted to reject non-UTF8 pathnames to cause similar failures. Our test suite needs to detect this environment and skip these tests there.

test_sqlite3 has multiple "undecodable path" tests that fail on ZFS. Marking the following issues as duplicates of this issue:

• Can't build 3.13b2 on Ubuntu: test test_sqlite3 failed #120191
• Python 3.13: test_sqlite3 fails. TESTFN_UNDECODABLE in test/support/os_helper.py is broken on zfs. #125117

ISTM we need a test support helper that can identify OS conditions such as this particular ZFS configuration. When that is in place, we can gate all "undecodable path" tests using this condition.

Temporary workarounds if you need to build python on system with ZFS:

1. Build without optimizations.

2. OR build with optimizations in tempfs.

mkdir buildpy
mount -t tmpfs -o size=512M tmpfs buildpy  # may require root
# download source code, in my case it's "Python-3.13.0.tgz"
tar xf /path/to/Python-3.13.0.tgz -C buildpy
cd buildpy/Python-3.13.0  # your path can differ
./configure --enable-optimizations  # your flags can differ
make -j16  # your flags can differ

At the end, there's ./python built with optimizations. This works because building is done in a different file system. Don't forget to umount buildpy directory.

If anyone else ever comes here from pyenv, just want to provide a little help.

1. Mostly follow Multiple test failures with OSError: [Errno 84] Invalid or incomplete multibyte or wide character on ZFS with utf8only=on #81765 (comment) (the above comment)
2. Add TMPDIR=/path/to/tmpfs-mount to your run command
   • e.g.:

    env TMPDIR='/path/to/tmpfs-mount' PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' PROFILE_TASK='-m test.regrtest --pgo -j0' pyenv install --verbose 3.13.4
   

Edit: you're also gonna need way more space than 512MB :)