Code itself aside, I don't think this PR solves the current issue.

Yes, with the prefetch argument, the executor will only schedule a certain amount of tasks in the beginning. However, if you enumerate the result iterator, it will just have the exact same effect - blocking the manager thread due to a large amount of submits.

Also, this design means the worker process won't work, until the user requests the result - which is very counter intuitive. If the user maps a large amount of data, they would hope that the data is being processed in the background by the executor, not wait there until the user asks for the result.

So, from my personal perspective, the design is not what we want for the executor. If you want to fix this, you need more than this.

Also the original issue is solvable by chunksize

Unfortunately chunksize is not available for ThreadPoolExecutor, which is best suited for I/O operations. Do you know of any better alternatives that can handle large iterators?

Unfortunately chunksize is not available for ThreadPoolExecutor, which is best suited for I/O operations. Do you know of any better alternatives that can handle large iterators?

I believe the reason ThreadPoolExecutor does not have the argument, is because thread pool does not suffer from it. There's no expensive submit wakeup pipe queue so you can do large iterators just fine.

Thanks for the discussion! Would love @graingert @rdarder @kumaraditya303 @brianquinlan your thoughts as well

So this could break backward compatibility.

I'm not sure whether I'm following fully here, but my intention was if we use the default or None for prefetch, we submit all tasks in the beginning. And I'm pretty sure that's how I implemented it, but I can very much likely have missed something, so very open to feedback here. It would be really useful to have an example here. @gaogaotiantian

Also the original issue is solvable by chunksize.

Do we mean the issue in bpo-29842 as well?

Unfortunately, I don't think this PR is worth more effort.

No worries at all, less work for me if that's case :)

I'm not sure whether I'm following fully here, but my intention was if we use the default or None for prefetch, we submit all tasks in the beginning. And I'm pretty sure that's how I implemented it, but I can very much likely have missed something, so very open to feedback here. It would be really useful to have an example here. @gaogaotiantian

My wording is inaccurate in that comment. What I really meant is - with the prefetch argument set, the iterables will not be resolved immediately, as compared to what the current docs states, which is definitely not the end of the world, especially considering prefetch itself expresses the meaning of "lazily". However, I would still be concerned because that makes the behavior more complicated that what it currently documented.

Do we mean the issue in bpo-29842 as well?

No, that one will not be solved by chunksize.

I think the fundamental factors we need beforep proceeding this, is the gain v cost. What do we get from the change?

• Making executing.map similar to map sounds great and consistent, but we can't have that. What we can have is an optional "less eager" solution. That's not the consistency we hoped.
• When will prefetch be used? Is there any benefit using it when the input is not infinite (compared to chunksize)? The current implementation only submits the task when it needs the result, which causes a delay in communication. So by doing it lazily it might take much more time to finish the iterable.
• If this only helps when the iterable is infinite, how common is that? What's the workaround and if that's acceptable?

I'm not saying no. Just how I consider this issue. This might be why the original issue has been sitting there for a couple of years - it might be a serious issue to the users.

Thanks for the comments! @gaogaotiantian

However, I would still be concerned because that makes the behavior more complicated that what it currently documented.

Happy to update the documentation if helpful.

When will prefetch be used? Is there any benefit using it when the input is not infinite (compared to chunksize)? The current implementation only submits the task when it needs the result, which causes a delay in communication. So by doing it lazily it might take much more time to finish the iterable.

I guess the main rationale is to give the users the flexibility, to choose how many items, from the input iterator, they would like to be processed at a time. This is certainly useful in the infinite iterator case, and I would imagine this to be useful in cases where, I don't want too many tasks to be processed at one time as well.
For example, I might be sending some requests to downstream services and I don't want that service to be overloaded. This will come in handy to be a simple rate-limiting mechanism. (Might be a poor example but I hope you get my point.)

Sorry but I still do not get the rationale for it. I understood your statements, but it did not convince me that this is a feature that worth the effort. The main concern is the usage - the current implementation will not submit task until the result is asked, after the prefetched ones, that just does not feel right to me.

You'll need some core dev behind this anyway (I'm not one). If you can find someone who likes the idea, you might be able to make some progress on this PR.

Hey @gpshead, sorry for tagging, but since this is concurrency related, I thought you might be interested

Hi, fyi here is a follow up PR: #125663 🙏🏻