Skip to content

Expose max_queue_size in ThreadPoolExecutor #73781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
prayerslayer mannequin opened this issue Feb 17, 2017 · 6 comments
Open

Expose max_queue_size in ThreadPoolExecutor #73781

prayerslayer mannequin opened this issue Feb 17, 2017 · 6 comments
Labels
3.8 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@prayerslayer
Copy link
Mannequin

prayerslayer mannequin commented Feb 17, 2017

BPO 29595
Nosy @rhettinger, @pitrou, @applio, @zhangyangyu, @tomMoral, @DimitrisJim, @prayerslayer, @stephenoneal, @iforapsy, @tianc777
PRs
  • bpo-29595: Expose max_queue_size in ThreadPoolExecutor #143
  • bpo-29595: Add queue size for pool. #23864
  • bpo-29595: Add queue size for pool. #23865
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2017-02-17.22:10:23.402>
labels = ['3.8', 'type-feature', 'library']
title = 'Expose max_queue_size in ThreadPoolExecutor'
updated_at = <Date 2021-10-14.21:59:47.298>
user = 'https://github.com/prayerslayer'

    bugs.python.org fields:

    activity = <Date 2021-10-14.21:59:47.298>
actor = 'iforapsy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-02-17.22:10:23.402>
creator = 'prayerslayer'
dependencies = []
files = []
hgrepos = []
issue_num = 29595
keywords = ['patch']
message_count = 6.0
messages = ['288043', '289082', '290253', '290949', '314699', '314743']
nosy_count = 11.0
nosy_names = ['rhettinger', 'pitrou', 'python-dev', 'davin', 'xiang.zhang', 'tomMoral', 'Jim Fasarakis-Hilliard', 'prayerslayer', 'stephen.oneal.04', 'iforapsy', 'tianc777']
pr_nums = ['143', '23864', '23865']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29595'
versions = ['Python 3.8']
    @prayerslayer
    Copy link
    Mannequin Author

    prayerslayer mannequin commented Feb 17, 2017

    Hi!

    I think the ThreadPoolExecutor should allow to set the maximum size of the underlying queue.

    The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API calls; I had to move data from one S3 bucket to another (~150M objects). Contrary to what I expected the maximum size of the underlying queue doesn't have a non-zero value by default. Thus my process ended up consuming gigabytes of memory, because it put more items into the queue than the threads were able to work off: The queue just kept growing. (It ran on K8s and the pod was rightfully killed eventually.)

    Of course there ways to work around this. One could use more threads, to some extent. Or you could use your own queue with a defined maximum size. But I think it's more work for users of Python than necessary.

    @prayerslayer prayerslayer mannequin added the type-feature A feature request or enhancement label Feb 17, 2017
    @Mariatta Mariatta added stdlib Python modules in the Lib dir 3.7 (EOL) end of life labels Feb 18, 2017
    @prayerslayer
    Copy link
    Mannequin Author

    prayerslayer mannequin commented Mar 6, 2017

    Hello again, there's a reviewed PR open for this issue and it hasn't even received authoritative feedback yet (ie whether or not you intend to support this feature at all). I would be very happy if a core dev could look over this change before everyone forgets about it :)

    @prayerslayer
    Copy link
    Mannequin Author

    prayerslayer mannequin commented Mar 24, 2017

    Ping. That's really a two-line change, can be easily reviewed in 15 minutes :)

    @rhettinger
    Copy link
    Contributor

    Prayslayer, please don't shove. Your PR request was responded to by Mariatta so it wasn't ignored.

    Making decisions about API expansions takes a while (making sure it fits the intended use, that it isn't a bug factory itself, that it is broadly useful, that it is the best solution to the problem, that is doesn't complicate the implementation or limit future opportunities, that there are unforeseen problems). Among the core developers, there are only a couple part-time contributors who are qualified to make these assessments for the multi-processing module (those devs don't include me).

    @stephenoneal
    Copy link
    Mannequin

    stephenoneal mannequin commented Mar 30, 2018

    My project we're going into the underlying _work_queue and blocking adding more elements based on unfinished_tasks to accomplish this, bubbling this up to the API would be a welcome addition.

    @pitrou
    Copy link
    Member

    pitrou commented Mar 31, 2018

    Please note the PR here has some review comments that need addressing.
    Also, it needs its conflicts with git master resolved.

    I'm cc'ing Thomas Moreau, who has done a lot of work recently on the concurrent.futures internals.

    @pitrou pitrou added 3.8 (EOL) end of life and removed 3.7 (EOL) end of life labels Mar 31, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    Threading issues 🧵
    Status: No status
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @applio @rhettinger @pitrou @Mariatta