Skip to content

WIP: [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) #9938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

WIP: [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) #9938

wants to merge 3 commits into from

Conversation

matrixise
Copy link
Member

@matrixise matrixise commented Oct 18, 2018 •
edited
Loading

On macOS, fix reading from and writing into a file with a size larger
than 2 GiB.

https://bugs.python.org/issue24658

@bedevere-bot bedevere-bot mentioned this pull request Oct 18, 2018
Merged
@matrixise matrixise changed the title [2.7] bpo-24568: Fix read/write greater than 2 GiB on macOS (GH-1705) [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) Oct 18, 2018
vstinner
vstinner requested changes Oct 18, 2018
View reviewed changes
Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should also patch:

  • Modules/_io/fileio.c: fileio_readinto(), fileio_readall(), fileio_read(), fileio_write()
  • Modules/posixmodule.c: posix_read(), posix_write()

You can add the #define at the end of Include/pyport.h.

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@matrixise
Copy link
Member Author

@vstinner In fact, I should mark this PR as "DO-NOT-MERGE" because I didn't finish it yesterday :/

@matrixise matrixise changed the title [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) WIP: [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) Oct 18, 2018
@matrixise matrixise changed the title WIP: [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) Oct 18, 2018
vstinner
vstinner requested changes Oct 18, 2018
View reviewed changes
Modules/posixmodule.c
@@ -6805,6 +6805,9 @@ posix_read(PyObject *self, PyObject *args)
return posix_error();
}
Py_BEGIN_ALLOW_THREADS
if (size > _PY_READ_MAX) {
size = _PY_READ_MAX;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum. It seems inefficient to allocate size bytes and only later pass _PY_READ_MAX. This truncation should be done before PyString_FromStringAndSize().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @matrixise

@serhiy-storchaka
Copy link
Member

Shouldn't we update also Py_UniversalNewlineFread()?

@vstinner
Copy link
Member

Shouldn't we update also Py_UniversalNewlineFread()?

It's implemented with fread() which uses size_t for the length parameter. I hope that the libc on macOS implements fread() properly for read larger than 2 GiB. If it's not the case, it's would be a bug in the libc, no? :-)

@vstinner
Copy link
Member

I hope that the libc on macOS implements fread() properly for read larger than 2 GiB.

This old issue https://bugs.python.org/issue9015#msg107966 explains that fwrite() on Windows should not be called with a buffer larger than 2**31 bytes. See also https://bugs.python.org/issue9611#msg120388

@matrixise matrixise changed the title [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) WIP: [2.7] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) Oct 25, 2018
@matrixise
Copy link
Member Author

ok, I will continue this PR, try to synthesize your comments and try to fix this PR. thanks

@matrixise
Copy link
Member Author

Need to finish this PR, I have macOS, will try to port the patch to Python 2.7.

Thank you for your patience and I am really sorry for the big delay 👎

@vstinner
Copy link
Member

Need to finish this PR, I have macOS, will try to port the patch to Python 2.7. Thank you for your patience and I am really sorry for the big delay -1

I don't understand the purpose of such comment. It sends me an unsolicited notification (email).

@zware
Copy link
Member

zware commented Jan 19, 2020

@matrixise EOL snuck up on you before this got finished, so I'm closing it for you :)

@zware zware closed this Jan 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vstinner vstinner vstinner requested changes

@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka

Assignees
No one assigned
Labels
awaiting changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@matrixise @bedevere-bot @serhiy-storchaka @vstinner @zware @the-knights-who-say-ni