Skip to content

Support out-of-band pickling for builtin types #84895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jakirkham mannequin opened this issue May 21, 2020 · 2 comments
Open

Support out-of-band pickling for builtin types #84895

jakirkham mannequin opened this issue May 21, 2020 · 2 comments
Labels
3.13 bugs and security fixes performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@jakirkham
Copy link
Mannequin

jakirkham mannequin commented May 21, 2020

BPO 40718
Nosy @pitrou, @jakirkham

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-05-21.19:25:54.076>
labels = ['3.8', '3.9', '3.10', 'performance', '3.11', 'library']
title = 'Support out-of-band pickling for builtin types'
updated_at = <Date 2021-09-30.00:30:58.504>
user = 'https://github.com/jakirkham'

bugs.python.org fields:

activity = <Date 2021-09-30.00:30:58.504>
actor = 'jakirkham'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-05-21.19:25:54.076>
creator = 'jakirkham'
dependencies = []
files = []
hgrepos = []
issue_num = 40718
keywords = []
message_count = 1.0
messages = ['369533']
nosy_count = 2.0
nosy_names = ['pitrou', 'jakirkham']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue40718'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']
@jakirkham
Copy link
Mannequin Author

jakirkham mannequin commented May 21, 2020

It would be nice (where possible) to support out-of-band pickling of builtin bytes-like types. This would allow binary data from these objects to be shipped along separately zero-copy and later reconstructed during unpickling.

It seems that bytes, bytearray, and array would be good candidates for this behavior. Not sure if mmap or memoryview would make sense as it might not be clear on how to reconstruct them during unpickling, but if someone sees a way those would be nice to support too.

To illustrate this a bit, here is the behavior with a bytes object today:

In [1]: import pickle

In [2]: b = b"abc"

In [3]: l = []

In [4]: p = pickle.dumps(b, protocol=5, buffer_callback=l.append)

In [5]: l
Out[5]: []

With this change, we would see this behavior instead:

In [1]: import pickle

In [2]: b = b"abc"

In [3]: l = []

In [4]: p = pickle.dumps(b, protocol=5, buffer_callback=l.append)

In [5]: l
Out[5]: [<pickle.PickleBuffer at 0x10efe7540>]

(This is my first Python bug submission. So apologies if I got turned around here. Please go easy on me :)

@jakirkham jakirkham mannequin added 3.8 (EOL) end of life 3.9 only security fixes performance Performance or resource usage labels May 21, 2020
@jakirkham jakirkham mannequin added 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir labels Sep 27, 2021
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@iritkatriel
Copy link
Member

See also #89467.

@iritkatriel iritkatriel added type-feature A feature request or enhancement 3.12 only security fixes and removed 3.11 only security fixes 3.10 only security fixes 3.9 only security fixes 3.8 (EOL) end of life labels Sep 11, 2022
@iritkatriel iritkatriel mentioned this issue Sep 11, 2022
Open
@erlend-aasland erlend-aasland added 3.13 bugs and security fixes and removed 3.12 only security fixes labels Jan 5, 2024
@picnixz picnixz mentioned this issue Jun 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.13 bugs and security fixes performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
Pickle and copy issues 🥒
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@iritkatriel @erlend-aasland