Skip to content

[WIP] GH-73991: Add os.copy() and friends #119079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[WIP] GH-73991: Add os.copy() and friends #119079

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
13e994b
GH-73991: Add `os.copyfile()` and friends
barneygale May 15, 2024
01afa3e
Move a few more tests over.
barneygale May 15, 2024
b3df117
Add `os.copy()`
barneygale May 15, 2024
ab09f78
Docs and tests fixes
barneygale May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
152 changes: 152 additions & 0 deletions Doc/library/os.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3862,6 +3862,158 @@ features:
.. versionadded:: 3.10


.. _os-copying-files:

Copying Files
~~~~~~~~~~~~~

Functions involving a file copy (:func:`copyfile` and :func:`copy`) may use
platform-specific "fast-copy" syscalls in order to copy the file more
efficiently (see :issue:`33671`).
"fast-copy" means that the copying operation occurs within the kernel, avoiding
the use of userspace buffers in Python as in "``outfd.write(infd.read())``".

On macOS `fcopyfile <http://www.manpagez.com/man/3/copyfile/>`_ is used to
copy the file content (not metadata).

On Linux :func:`os.sendfile` is used.

On Windows :func:`copyfile` uses a bigger default buffer size (1 MiB
instead of 64 KiB) and a :func:`memoryview`-based variant of
:func:`copyfileobj` is used.

If the fast-copy operation fails and no data was written in the destination
file then shutil will silently fallback on using less efficient
:func:`copyfileobj` function internally.


.. exception:: SameFileError

This exception is raised if source and destination in :func:`copyfile`
are the same file.

.. versionadded:: 3.14


.. function:: copyfileobj(fsrc, fdst[, length])

Copy the contents of the :term:`file-like object <file object>` *fsrc* to
the file-like object *fdst*. The integer *length*, if given, is the buffer
size. In particular, a negative *length* value means to copy the data
without looping over the source data in chunks; by default the data is read
in chunks to avoid uncontrolled memory consumption. Note that if the
current file position of the *fsrc* object is not 0, only the contents from
the current file position to the end of the file will be copied.

.. versionadded:: 3.14


.. function:: copyfile(src, dst, *, follow_symlinks=True)

Copy the contents (no metadata) of the file named *src* to a file named
*dst* and return *dst* in the most efficient way possible.
*src* and *dst* are :term:`path-like objects <path-like object>` or path
names given as strings.

*dst* must be the complete target file name; look at :func:`shutil.copy`
for a copy that accepts a target directory path. If *src* and *dst*
specify the same file, :exc:`SameFileError` is raised.

The destination location must be writable; otherwise, an :exc:`OSError`
exception will be raised. If *dst* already exists, it will be replaced.
Special files such as character or block devices and pipes cannot be
copied with this function.

If *follow_symlinks* is false and *src* is a symbolic link,
a new symbolic link will be created instead of copying the
file *src* points to.

.. audit-event:: os.copyfile src,dst os.copyfile

.. versionadded:: 3.14


.. function:: copymode(src, dst, *, follow_symlinks=True)

Copy the permission bits from *src* to *dst*. The file contents, owner, and
group are unaffected. *src* and *dst* are
:term:`path-like objects <path-like object>` or path names given as
strings. If *follow_symlinks* is false, and both *src* and *dst* are
symbolic links, :func:`copymode` will attempt to modify the mode of *dst*
itself (rather than the file it points to). This functionality is not
available on every platform; please see :func:`copystat` for more
information. If :func:`copymode` cannot modify symbolic links on the local
platform, and it is asked to do so, it will do nothing and return.

.. audit-event:: os.copymode src,dst os.copymode

.. versionadded:: 3.14


.. function:: copystat(src, dst, *, follow_symlinks=True)

Copy the permission bits, last access time, last modification time, and
flags from *src* to *dst*. On Linux, :func:`copystat` also copies the
"extended attributes" where possible. The file contents, owner, and group
are unaffected. *src* and *dst* are
:term:`path-like objects <path-like object>` or path names given as
strings.

If *follow_symlinks* is false, and *src* and *dst* both refer to symbolic
links, :func:`copystat` will operate on the symbolic links themselves
rather than the files the symbolic links refer to—reading the information
from the *src* symbolic link, and writing the information to the *dst*
symbolic link.

.. note::

Not all platforms provide the ability to examine and modify symbolic
links. Python itself can tell you what functionality is locally
available.

* If ``os.chmod in os.supports_follow_symlinks`` is ``True``,
:func:`copystat` can modify the permission bits of a symbolic link.

* If ``os.utime in os.supports_follow_symlinks`` is ``True``,
:func:`copystat` can modify the last access and modification times of
a symbolic link.

* If ``os.chflags in os.supports_follow_symlinks`` is ``True``,
:func:`copystat` can modify the flags of a symbolic link.
(``os.chflags`` is not available on all platforms.)

On platforms where some or all of this functionality is unavailable,
when asked to modify a symbolic link, :func:`copystat` will copy
everything it can. :func:`copystat` never returns failure.

Please see :data:`supports_follow_symlinks` for more information.

.. audit-event:: os.copystat src,dst os.copystat

.. versionadded:: 3.14


.. function:: copy(src, dst, *, follow_symlinks=True)

Copies the file *src* to the file *dst*. *src* and *dst* should be
:term:`path-like objects <path-like object>` or strings. If *dst* specifies
a file that already exists, it will be replaced.

When *follow_symlinks* is false, and *src* is a symbolic link, :func:`copy`
attempts to copy all metadata from the *src* symbolic link to the newly
created *dst* symbolic link. However, this functionality is not available
on all platforms. On platforms where some or all of this functionality is
unavailable, :func:`copy` will preserve all the metadata it can;
:func:`copy` never raises an exception because it cannot preserve file
metadata.

:func:`copy` uses :func:`copystat` to copy the file metadata. Please see
:func:`copystat` for more information about platform support for modifying
symbolic link metadata.

.. versionadded:: 3.14


Timer File Descriptors
~~~~~~~~~~~~~~~~~~~~~~

Expand Down
128 changes: 12 additions & 116 deletions Doc/library/shutil.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,35 +39,12 @@ Directory and files operations

.. function:: copyfileobj(fsrc, fdst[, length])

Copy the contents of the :term:`file-like object <file object>` *fsrc* to the file-like object *fdst*.
The integer *length*, if given, is the buffer size. In particular, a negative
*length* value means to copy the data without looping over the source data in
chunks; by default the data is read in chunks to avoid uncontrolled memory
consumption. Note that if the current file position of the *fsrc* object is not
0, only the contents from the current file position to the end of the file will
be copied.
Alias of :func:`os.copyfileobj`.


.. function:: copyfile(src, dst, *, follow_symlinks=True)

Copy the contents (no metadata) of the file named *src* to a file named
*dst* and return *dst* in the most efficient way possible.
*src* and *dst* are :term:`path-like objects <path-like object>` or path names given as strings.

*dst* must be the complete target file name; look at :func:`~shutil.copy`
for a copy that accepts a target directory path. If *src* and *dst*
specify the same file, :exc:`SameFileError` is raised.

The destination location must be writable; otherwise, an :exc:`OSError`
exception will be raised. If *dst* already exists, it will be replaced.
Special files such as character or block devices and pipes cannot be
copied with this function.

If *follow_symlinks* is false and *src* is a symbolic link,
a new symbolic link will be created instead of copying the
file *src* points to.

.. audit-event:: shutil.copyfile src,dst shutil.copyfile
Alias of :func:`os.copyfile`.

.. versionchanged:: 3.3
:exc:`IOError` used to be raised instead of :exc:`OSError`.
Expand All @@ -80,77 +57,25 @@ Directory and files operations

.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
copy the file more efficiently. See :ref:`os-copying-files`.

.. exception:: SameFileError

This exception is raised if source and destination in :func:`copyfile`
are the same file.
Alias of :exc:`os.SameFileError`.

.. versionadded:: 3.4


.. function:: copymode(src, dst, *, follow_symlinks=True)

Copy the permission bits from *src* to *dst*. The file contents, owner, and
group are unaffected. *src* and *dst* are :term:`path-like objects <path-like object>` or path names
given as strings.
If *follow_symlinks* is false, and both *src* and *dst* are symbolic links,
:func:`copymode` will attempt to modify the mode of *dst* itself (rather
than the file it points to). This functionality is not available on every
platform; please see :func:`copystat` for more information. If
:func:`copymode` cannot modify symbolic links on the local platform, and it
is asked to do so, it will do nothing and return.

.. audit-event:: shutil.copymode src,dst shutil.copymode
Alias of :func:`os.copymode`.

.. versionchanged:: 3.3
Added *follow_symlinks* argument.

.. function:: copystat(src, dst, *, follow_symlinks=True)

Copy the permission bits, last access time, last modification time, and
flags from *src* to *dst*. On Linux, :func:`copystat` also copies the
"extended attributes" where possible. The file contents, owner, and
group are unaffected. *src* and *dst* are :term:`path-like objects <path-like object>` or path
names given as strings.

If *follow_symlinks* is false, and *src* and *dst* both
refer to symbolic links, :func:`copystat` will operate on
the symbolic links themselves rather than the files the
symbolic links refer to—reading the information from the
*src* symbolic link, and writing the information to the
*dst* symbolic link.

.. note::

Not all platforms provide the ability to examine and
modify symbolic links. Python itself can tell you what
functionality is locally available.

* If ``os.chmod in os.supports_follow_symlinks`` is
``True``, :func:`copystat` can modify the permission
bits of a symbolic link.

* If ``os.utime in os.supports_follow_symlinks`` is
``True``, :func:`copystat` can modify the last access
and modification times of a symbolic link.

* If ``os.chflags in os.supports_follow_symlinks`` is
``True``, :func:`copystat` can modify the flags of
a symbolic link. (``os.chflags`` is not available on
all platforms.)

On platforms where some or all of this functionality
is unavailable, when asked to modify a symbolic link,
:func:`copystat` will copy everything it can.
:func:`copystat` never returns failure.

Please see :data:`os.supports_follow_symlinks`
for more information.

.. audit-event:: shutil.copystat src,dst shutil.copystat
Alias of :func:`os.copystat`.

.. versionchanged:: 3.3
Added *follow_symlinks* argument and support for Linux extended attributes.
Expand Down Expand Up @@ -184,13 +109,13 @@ Directory and files operations

.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
copy the file more efficiently. See :ref:`os-copying-files`.

.. function:: copy2(src, dst, *, follow_symlinks=True)

Identical to :func:`~shutil.copy` except that :func:`copy2`
also attempts to preserve file metadata.
also attempts to preserve file metadata. Identical to
:func:`os.copy` except that *dst* may be a directory.

When *follow_symlinks* is false, and *src* is a symbolic
link, :func:`copy2` attempts to copy all metadata from the
Expand All @@ -216,8 +141,7 @@ Directory and files operations

.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
copy the file more efficiently. See :ref:`os-copying-files`.

.. function:: ignore_patterns(*patterns)

Expand Down Expand Up @@ -286,8 +210,7 @@ Directory and files operations

.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
copy the file more efficiently. See :ref:`os-copying-files`.

.. versionchanged:: 3.8
Added the *dirs_exist_ok* parameter.
Expand Down Expand Up @@ -395,8 +318,7 @@ Directory and files operations

.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
copy the file more efficiently. See :ref:`os-copying-files`.

.. versionchanged:: 3.9
Accepts a :term:`path-like object` for both *src* and *dst*.
Expand Down Expand Up @@ -502,32 +424,6 @@ Directory and files operations
operation. For :func:`copytree`, the exception argument is a list of 3-tuples
(*srcname*, *dstname*, *exception*).

.. _shutil-platform-dependent-efficient-copy-operations:

Platform-dependent efficient copy operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Starting from Python 3.8, all functions involving a file copy
(:func:`copyfile`, :func:`~shutil.copy`, :func:`copy2`,
:func:`copytree`, and :func:`move`) may use
platform-specific "fast-copy" syscalls in order to copy the file more
efficiently (see :issue:`33671`).
"fast-copy" means that the copying operation occurs within the kernel, avoiding
the use of userspace buffers in Python as in "``outfd.write(infd.read())``".

On macOS `fcopyfile`_ is used to copy the file content (not metadata).

On Linux :func:`os.sendfile` is used.

On Windows :func:`shutil.copyfile` uses a bigger default buffer size (1 MiB
instead of 64 KiB) and a :func:`memoryview`-based variant of
:func:`shutil.copyfileobj` is used.

If the fast-copy operation fails and no data was written in the destination
file then shutil will silently fallback on using less efficient
:func:`copyfileobj` function internally.

.. versionchanged:: 3.8

.. _shutil-copytree-example:

Expand Down
5 changes: 2 additions & 3 deletions Doc/whatsnew/3.8.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1481,7 +1481,7 @@ Optimizations
The speedup for copying a 512 MiB file within the same partition is about
+26% on Linux, +50% on macOS and +40% on Windows. Also, much less CPU cycles
are consumed.
See :ref:`shutil-platform-dependent-efficient-copy-operations` section.
See :ref:`os-copying-files`.
(Contributed by Giampaolo Rodolà in :issue:`33671`.)

* :func:`shutil.copytree` uses :func:`os.scandir` function and all copy
Expand Down Expand Up @@ -1921,8 +1921,7 @@ Changes in the Python API

* :func:`shutil.copyfile`, :func:`shutil.copy`, :func:`shutil.copy2`,
:func:`shutil.copytree` and :func:`shutil.move` use platform-specific
"fast-copy" syscalls (see
:ref:`shutil-platform-dependent-efficient-copy-operations` section).
"fast-copy" syscalls (see :ref:`os-copying-files`).

* :func:`shutil.copyfile` default buffer size on Windows was changed from
16 KiB to 1 MiB.
Expand Down
Loading
Loading