Skip to content

gh-117587: Add C implementation of os.path.abspath #117855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 87 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
2475af9
Speedup `posixpath.abspath()` for relative paths
nineteendo Apr 12, 2024
807b494
Fix cwd length for byte paths & bus error
nineteendo Apr 12, 2024
aa3ed0c
Merge branch 'python:main' into speedup-posixpath.abspath
nineteendo Apr 12, 2024
fa50a4c
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 13, 2024
f502dd1
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 13, 2024
716f594
Fix `_Py_normpath_and_size` when `size=-1`
nineteendo Apr 13, 2024
cc3a59c
Update Modules/posixmodule.c
nineteendo Apr 13, 2024
f5f2c7b
Update Python/fileutils.c
nineteendo Apr 13, 2024
a1bc125
fastcall & unicode type
nineteendo Apr 13, 2024
ae2c040
Revert changes to `os__path_normpath_impl`
nineteendo Apr 14, 2024
95773d9
simply `os.fsdecode(os.getcwdb())`
nineteendo Apr 14, 2024
b77a15d
1st attempt at full C implementation
nineteendo Apr 14, 2024
f39ab83
Merge branch 'speedup-posixpath.abspath' of https://github.com/ninete…
nineteendo Apr 14, 2024
c63518b
Update Modules/posixmodule.c
nineteendo Apr 14, 2024
94b12c2
2nd attempt at full C implementation
nineteendo Apr 14, 2024
c6c62d2
Merge branch 'speedup-posixpath.abspath' of https://github.com/ninete…
nineteendo Apr 14, 2024
51c9541
3rd attempt at full C implementation
nineteendo Apr 14, 2024
ea0b4d1
Fix tests
nineteendo Apr 14, 2024
8d435d6
Fix tests attempt 2
nineteendo Apr 14, 2024
7057f3b
special case for '' & '.'
nineteendo Apr 15, 2024
fff561e
follow pep 7
nineteendo Apr 15, 2024
15c8f0d
check type with argument clinic
nineteendo Apr 15, 2024
34858a8
Fix memory leak & variable names
nineteendo Apr 16, 2024
21cd24a
Use `PyMem_RawFree`
nineteendo Apr 16, 2024
1113ec3
Update Modules/posixmodule.c
nineteendo Apr 16, 2024
7cca80e
Update Modules/posixmodule.c
nineteendo Apr 16, 2024
0f1f28b
Merge branch 'main' into speedup-posixpath.abspath
eryksun Apr 16, 2024
1313ceb
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 16, 2024
7b6727c
Update Modules/posixmodule.c
nineteendo Apr 17, 2024
97516f6
narrow scoping
nineteendo Apr 17, 2024
7e87ccd
Free `cwd_buf` for all exits
nineteendo Apr 17, 2024
60c99be
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 17, 2024
c06d34f
rename `start` to `prefix_len`
nineteendo Apr 17, 2024
ba14b7d
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 18, 2024
3d2836e
test different cache key
nineteendo Apr 21, 2024
9fcedb5
Merge branch 'main' into speedup-posixpath.abspath
hugovk Apr 23, 2024
111df43
Add C implementation of `ntpath.abspath()`
nineteendo Apr 23, 2024
a538218
Replace with `nt._path_abspath`
nineteendo Apr 23, 2024
d36379d
Update Lib/posixpath.py
nineteendo Apr 23, 2024
6e645f1
Remove fallback
nineteendo Apr 24, 2024
8fd2984
Update Modules/posixmodule.c
nineteendo Apr 24, 2024
4d8eb1d
Follow pep 7
nineteendo Apr 24, 2024
c470123
Check for embedded null characters
nineteendo Apr 25, 2024
3a9af3f
Stricter check for embedded null characters
nineteendo Apr 25, 2024
329692b
Merge branch 'main' into speedup-os.path
nineteendo Apr 25, 2024
9fbc62e
Support qualified referencing
nineteendo Apr 25, 2024
d83a580
Add additional test
nineteendo Apr 25, 2024
08b9674
null in cwd?
nineteendo Apr 25, 2024
6eb1442
Add Windows 11 check
nineteendo Apr 25, 2024
b7f9df6
Add additional test
nineteendo Apr 25, 2024
b127f05
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 26, 2024
3ad0d7f
Fix drive relative paths
nineteendo Apr 26, 2024
549fa4e
Handle all qualified referencing
nineteendo Apr 27, 2024
a768039
Fix ambiguous operation
nineteendo Apr 27, 2024
3339c19
Special case for '.'
nineteendo Apr 27, 2024
0a40599
Handle drive relative paths
nineteendo Apr 28, 2024
2315bbb
Add tests
nineteendo Apr 28, 2024
bbbba83
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 28, 2024
6bac80e
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 30, 2024
b3d04ac
Direct C call for `nt.isdevdrive()`
nineteendo May 4, 2024
415f831
Merge branch 'main' into speedup-posixpath.abspath
nineteendo May 4, 2024
9c55537
Revert "Direct C call for `nt.isdevdrive()`"
nineteendo May 4, 2024
2bf2cd3
Revert "Revert "Direct C call for `nt.isdevdrive()`""
nineteendo May 4, 2024
824f0d2
Restore lost code
nineteendo May 4, 2024
5850c3b
Zero-valued flag enum has no name
nineteendo May 9, 2024
eb26e22
Revert "Zero-valued flag enum has no name"
nineteendo May 9, 2024
bfb235c
Merge branch 'main' into speedup-posixpath.abspath
nineteendo May 27, 2024
d89e875
Direct C call for `posixpath.abspath()`
nineteendo May 28, 2024
66c057f
Optimisation
nineteendo May 28, 2024
8c1e11c
Use root directory for other drives
nineteendo May 28, 2024
50f5882
Fix drive relative paths
nineteendo May 28, 2024
74d7431
Simplify implementation
nineteendo May 28, 2024
807f83f
Update Lib/ntpath.py
nineteendo May 28, 2024
b19bbeb
Improve readability
nineteendo May 28, 2024
f934977
Check embedded null manually
nineteendo May 28, 2024
cf01b90
Revert fallback code on POSIX
nineteendo May 29, 2024
29f9cfc
Update Lib/ntpath.py
nineteendo May 29, 2024
374a39e
Update Lib/test/test_ntpath.py
nineteendo May 29, 2024
adc708d
Apply suggestions from code review
nineteendo May 29, 2024
4d0289a
Narrow scope
nineteendo May 29, 2024
fcffc53
Improve comments
nineteendo May 29, 2024
1ebe5f4
Just check if path is absolute
nineteendo May 29, 2024
e047f03
Narrow scope of `cwd_buf`
nineteendo May 30, 2024
625fc79
Update Python/fileutils.c
nineteendo May 30, 2024
fc3f227
Free memory before calling the `PyErr` APIs
nineteendo May 30, 2024
54c7021
Apply suggestions from code review
nineteendo May 30, 2024
dc66ef9
Use explicit return values
nineteendo May 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Include/internal/pycore_fileutils.h
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,8 @@ extern size_t _Py_find_basename(const wchar_t *filename);
// Export for '_testinternalcapi' shared extension
PyAPI_FUNC(wchar_t*) _Py_normpath(wchar_t *path, Py_ssize_t size);

extern wchar_t *_Py_normpath_and_size(wchar_t *path, Py_ssize_t size, Py_ssize_t *length);
extern wchar_t *_Py_normpath_and_size(wchar_t *path, Py_ssize_t size, Py_ssize_t start,
Py_ssize_t *length, int explicit_curdir);

// The Windows Games API family does not provide these functions
// so provide our own implementations. Remove them in case they get added
Expand Down
2 changes: 1 addition & 1 deletion Lib/ctypes/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ def __init__(self, name, mode=DEFAULT_MODE, handle=None,
import nt
mode = nt._LOAD_LIBRARY_SEARCH_DEFAULT_DIRS
if '/' in name or '\\' in name:
self._name = nt._getfullpathname(self._name)
self._name = nt._path_abspath(self._name)
mode |= nt._LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR

class _FuncPtr(_CFuncPtr):
Expand Down
51 changes: 31 additions & 20 deletions Lib/ntpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -554,36 +554,47 @@ def normpath(path):
return prefix + sep.join(comps)


def _abspath_fallback(path):
"""Return the absolute version of a path as a fallback function in case
`nt._getfullpathname` is not available or raises OSError. See bpo-31047 for
more.

"""

path = os.fspath(path)
if not isabs(path):
if isinstance(path, bytes):
cwd = os.getcwdb()
else:
cwd = os.getcwd()
path = join(cwd, path)
return normpath(path)

# Return an absolute path.
try:
from nt import _getfullpathname
from nt import _path_abspath

except ImportError: # not running on Windows - mock up something sensible
abspath = _abspath_fallback
def abspath(path):
"""Return the absolute version of a path."""
path = os.fspath(path)
if not isabs(path):
if isinstance(path, bytes):
path = join(os.getcwdb(), path)
else:
path = join(os.getcwd(), path)
return normpath(path)

else: # use native Windows method on Windows
def abspath(path):
"""Return the absolute version of a path."""
try:
return _getfullpathname(normpath(path))
return _path_abspath(path)
except (OSError, ValueError):
return _abspath_fallback(path)
# See gh-75230, handle outside for cleaner traceback
pass
path = os.fspath(path)
if not isabs(path):
if isinstance(path, bytes):
sep = b'/'
cwd = os.getcwdb()
else:
sep = '/'
cwd = os.getcwd()
drive, root, path = splitroot(path)
if drive and drive != splitroot(cwd)[0]:
try:
path = join(_path_abspath(drive), path)
except (OSError, ValueError):
# Invalid drive \x00: on Windows; assume root directory
path = drive + sep + path
else:
path = join(cwd, root + path)
return normpath(path)

try:
from nt import _findfirstfile, _getfinalpathname, readlink as _nt_readlink
Expand Down
23 changes: 13 additions & 10 deletions Lib/posixpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -372,16 +372,19 @@ def normpath(path):
return path or dot


def abspath(path):
"""Return an absolute path."""
path = os.fspath(path)
if isinstance(path, bytes):
if not path.startswith(b'/'):
path = join(os.getcwdb(), path)
else:
if not path.startswith('/'):
path = join(os.getcwd(), path)
return normpath(path)
try:
from posix import _path_abspath as abspath
except ImportError:
def abspath(path):
"""Return an absolute path."""
path = os.fspath(path)
if isinstance(path, bytes):
if not path.startswith(b'/'):
path = join(os.getcwdb(), path)
else:
if not path.startswith('/'):
path = join(os.getcwd(), path)
return normpath(path)


# Return a canonical path (i.e. the absolute location of a file on the
Expand Down
4 changes: 2 additions & 2 deletions Lib/test/test_ctypes/test_loading.py
Original file line number Diff line number Diff line change
Expand Up @@ -185,11 +185,11 @@ def should_fail(command):
should_pass("windll.kernel32.SetDllDirectoryW(None); WinDLL('_sqlite3.dll', winmode=0)")

# Full path load without DLL_LOAD_DIR shouldn't find dependency
should_fail("WinDLL(nt._getfullpathname('_sqlite3.dll'), " +
should_fail("WinDLL(nt._path_abspath('_sqlite3.dll'), " +
"winmode=nt._LOAD_LIBRARY_SEARCH_SYSTEM32)")

# Full path load with DLL_LOAD_DIR should succeed
should_pass("WinDLL(nt._getfullpathname('_sqlite3.dll'), " +
should_pass("WinDLL(nt._path_abspath('_sqlite3.dll'), " +
"winmode=nt._LOAD_LIBRARY_SEARCH_SYSTEM32|" +
"nt._LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR)")

Expand Down
19 changes: 19 additions & 0 deletions Lib/test/test_ntpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -347,13 +347,15 @@ def test_normpath(self):

tester("ntpath.normpath('..')", r'..')
tester("ntpath.normpath('.')", r'.')
tester("ntpath.normpath('c:.')", 'c:')
tester("ntpath.normpath('')", r'.')
tester("ntpath.normpath('/')", '\\')
tester("ntpath.normpath('c:/')", 'c:\\')
tester("ntpath.normpath('/../.././..')", '\\')
tester("ntpath.normpath('c:/../../..')", 'c:\\')
tester("ntpath.normpath('../.././..')", r'..\..\..')
tester("ntpath.normpath('K:../.././..')", r'K:..\..\..')
tester("ntpath.normpath('./a/b')", r'a\b')
tester("ntpath.normpath('C:////a/b')", r'C:\a\b')
tester("ntpath.normpath('//machine/share//a/b')", r'\\machine\share\a\b')

Expand Down Expand Up @@ -806,6 +808,9 @@ def test_abspath(self):
tester('ntpath.abspath("C:\\spam. . .")', "C:\\spam")
tester('ntpath.abspath("C:/nul")', "\\\\.\\nul")
tester('ntpath.abspath("C:\\nul")', "\\\\.\\nul")
self.assertTrue(ntpath.isabs(ntpath.abspath("C:spam")))
self.assertTrue(ntpath.isabs(ntpath.abspath("C:\x00")))
self.assertTrue(ntpath.isabs(ntpath.abspath("\x00:spam")))
tester('ntpath.abspath("//..")', "\\\\")
tester('ntpath.abspath("//../")', "\\\\..\\")
tester('ntpath.abspath("//../..")', "\\\\..\\")
Expand Down Expand Up @@ -836,6 +841,20 @@ def test_abspath(self):
tester('ntpath.abspath("")', cwd_dir)
tester('ntpath.abspath(" ")', cwd_dir + "\\ ")
tester('ntpath.abspath("?")', cwd_dir + "\\?")
tester('ntpath.abspath("con")', r"\\.\con")
# bpo-45354: Windows 11 changed MS-DOS device name handling
if sys.getwindowsversion()[:3] < (10, 0, 22000):
tester('ntpath.abspath("./con")', r"\\.\con")
tester('ntpath.abspath("foo/../con")', r"\\.\con")
tester('ntpath.abspath("con/foo/..")', r"\\.\con")
tester('ntpath.abspath("con/.")', r"\\.\con")
else:
tester('ntpath.abspath("./con")', cwd_dir + r"\con")
tester('ntpath.abspath("foo/../con")', cwd_dir + r"\con")
tester('ntpath.abspath("con/foo/..")', cwd_dir + r"\con")
tester('ntpath.abspath("con/.")', cwd_dir + r"\con")
tester('ntpath.abspath("./Z:spam")', cwd_dir + r"\Z:spam")
tester('ntpath.abspath("spam/../Z:eggs")', cwd_dir + r"\Z:eggs")
drive, _ = ntpath.splitdrive(cwd_dir)
tester('ntpath.abspath("/abc/")', drive + "\\abc")

Expand Down
33 changes: 20 additions & 13 deletions Lib/test/test_posixpath.py
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,8 @@ def test_fast_paths_in_use(self):
self.assertFalse(inspect.isfunction(os.path.splitroot))
self.assertTrue(os.path.normpath is posix._path_normpath)
self.assertFalse(inspect.isfunction(os.path.normpath))
self.assertTrue(os.path.abspath is posix._path_abspath)
self.assertFalse(inspect.isfunction(os.path.abspath))

def test_expanduser(self):
self.assertEqual(posixpath.expanduser("foo"), "foo")
Expand Down Expand Up @@ -382,6 +384,7 @@ def test_expanduser_pwd2(self):
("///..//./foo/.//bar", "/foo/bar"),
(".", "."),
(".//.", "."),
("./foo/bar", "foo/bar"),
("..", ".."),
("../", ".."),
("../foo", "../foo"),
Expand Down Expand Up @@ -689,19 +692,21 @@ def test_realpath_unreadable_symlink(self):
os.chmod(ABSTFN, 0o755, follow_symlinks=False)
os.unlink(ABSTFN)

@skip_if_ABSTFN_contains_backslash
def test_relpath(self):
(real_getcwd, os.getcwd) = (os.getcwd, lambda: r"/home/user/bar")
try:
curdir = os.path.split(os.getcwd())[-1]
os.mkdir(ABSTFN)
self.assertRaises(TypeError, posixpath.relpath, None)
self.assertRaises(ValueError, posixpath.relpath, "")
self.assertEqual(posixpath.relpath("a"), "a")
self.assertEqual(posixpath.relpath(posixpath.abspath("a")), "a")
self.assertEqual(posixpath.relpath("a/b"), "a/b")
self.assertEqual(posixpath.relpath("../a/b"), "../a/b")
self.assertEqual(posixpath.relpath("a", "../b"), "../"+curdir+"/a")
self.assertEqual(posixpath.relpath("a/b", "../c"),
"../"+curdir+"/a/b")
with os_helper.change_cwd(ABSTFN):
curdir = basename(ABSTFN)
self.assertEqual(posixpath.relpath("a", "../b"), "../"+curdir+"/a")
self.assertEqual(posixpath.relpath("a/b", "../c"),
"../"+curdir+"/a/b")
self.assertEqual(posixpath.relpath("a", "b/c"), "../../a")
self.assertEqual(posixpath.relpath("a", "a"), ".")
self.assertEqual(posixpath.relpath("/foo/bar/bat", "/x/y/z"), '../../../foo/bar/bat')
Expand All @@ -714,21 +719,23 @@ def test_relpath(self):
self.assertEqual(posixpath.relpath("/a", "/a"), '.')
self.assertEqual(posixpath.relpath("/a/b", "/a/b"), '.')
finally:
os.getcwd = real_getcwd
safe_rmdir(ABSTFN)

@skip_if_ABSTFN_contains_backslash
def test_relpath_bytes(self):
(real_getcwdb, os.getcwdb) = (os.getcwdb, lambda: br"/home/user/bar")
try:
curdir = os.path.split(os.getcwdb())[-1]
os.mkdir(ABSTFN)
self.assertRaises(ValueError, posixpath.relpath, b"")
self.assertEqual(posixpath.relpath(b"a"), b"a")
self.assertEqual(posixpath.relpath(posixpath.abspath(b"a")), b"a")
self.assertEqual(posixpath.relpath(b"a/b"), b"a/b")
self.assertEqual(posixpath.relpath(b"../a/b"), b"../a/b")
self.assertEqual(posixpath.relpath(b"a", b"../b"),
b"../"+curdir+b"/a")
self.assertEqual(posixpath.relpath(b"a/b", b"../c"),
b"../"+curdir+b"/a/b")
with os_helper.change_cwd(ABSTFN):
curdir = os.fsencode(basename(ABSTFN))
self.assertEqual(posixpath.relpath(b"a", b"../b"),
b"../"+curdir+b"/a")
self.assertEqual(posixpath.relpath(b"a/b", b"../c"),
b"../"+curdir+b"/a/b")
self.assertEqual(posixpath.relpath(b"a", b"b/c"), b"../../a")
self.assertEqual(posixpath.relpath(b"a", b"a"), b".")
self.assertEqual(posixpath.relpath(b"/foo/bar/bat", b"/x/y/z"), b'../../../foo/bar/bat')
Expand All @@ -744,7 +751,7 @@ def test_relpath_bytes(self):
self.assertRaises(TypeError, posixpath.relpath, b"bytes", "str")
self.assertRaises(TypeError, posixpath.relpath, "str", b"bytes")
finally:
os.getcwdb = real_getcwdb
safe_rmdir(ABSTFN)

def test_commonpath(self):
def check(paths, expected):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Speedup :func:`os.path.abspath` with a native implementation.
69 changes: 31 additions & 38 deletions Modules/clinic/posixmodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading