Skip to content

REPL: AttributeError: module __mp_main__ has no attribute is_prime in ProcessPoolExecutor example #132898

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wenming21 opened this issue Apr 25, 2025 · 7 comments
Open

REPL: AttributeError: module __mp_main__ has no attribute is_prime in ProcessPoolExecutor example #132898

wenming21 opened this issue Apr 25, 2025 · 7 comments
Labels
3.14 new features, bugs and security fixes stdlib Python modules in the Lib dir topic-multiprocessing type-bug An unexpected behavior, bug, or error

Comments

@wenming21
Copy link

wenming21 commented Apr 25, 2025 •
edited
Loading

Bug report

Bug description:

python version:3.12.9
An error occurred when running as a standalone script.
An error message is reported when running the sample code of the python document:

Traceback (most recent call last):
File "C:\Users\wen\Desktop\test.py", line 32, in
main()
File "C:\Users\wen\Desktop\test.py", line 28, in main
for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures\process.py", line 636, in _chain_from_iterable_of_lists
for element in iterable:
^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

import concurrent.futures
import math

PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]

def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False

    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True

def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
    main()

CPython versions tested on:

3.12

Operating systems tested on:

Windows 10
@wenming21 wenming21 added the type-bug An unexpected behavior, bug, or error label Apr 25, 2025
@picnixz picnixz added topic-multiprocessing stdlib Python modules in the Lib dir labels Apr 25, 2025
@picnixz
Copy link
Member

picnixz commented Apr 25, 2025 •
edited
Loading

Question: did this occur because you hit Ctrl+C or killed the process somehow?

Nevermind, it occurred on Linux as well with just running the example.

@picnixz picnixz changed the title A process in the process pool was terminated abruptly while the future was running or pending AttributeError: module __mp_main__ has no attribute is_prime in ProcessPoolExecutor example Apr 25, 2025
@picnixz picnixz changed the title AttributeError: module __mp_main__ has no attribute is_prime in ProcessPoolExecutor example REPL: AttributeError: module __mp_main__ has no attribute is_prime in ProcessPoolExecutor example Apr 25, 2025
@picnixz
Copy link
Member

picnixz commented Apr 25, 2025 •
edited
Loading

Ok, I only found the issue when the above code is being tested inside the REPL and not as a script (namely python test.py should work but not when I C/C the above code). The best fix for now would be to just indicate in the docs "run it as a standalone script, not inside the REPL", but I don't know the exact reason why (well I can guess: it has something with pickling and probably globals that are not made available because they are only available in the REPL and ProcessPoolExecutor is not able to find the correct "main" module).

cc @pablogsal @gpshead

@picnixz
Copy link
Member

picnixz commented Apr 25, 2025

@wenming21 Your latest edit indicates that the error also occurs when running the example as a standalone script. What is the full traceback? (namely, what's above the "Traceback (most recent call last):"? because I do have other stuff written when I encounter the bug)

@StanFromIreland
Copy link
Contributor

Why was this closed? @wenming21

@StanFromIreland
Copy link
Contributor

I can also add that this only occurs on the main branch, on 3.13 it works fine.

@picnixz picnixz reopened this Apr 25, 2025
@picnixz picnixz added the 3.14 new features, bugs and security fixes label Apr 25, 2025
@picnixz
Copy link
Member

picnixz commented Apr 25, 2025

3.14 has seen changes with the default start method so this could be the cause by the way (at least on Linux)

@YvesDup
Copy link
Contributor

YvesDup commented Apr 30, 2025

What's happen when you change the start_method ?
I am on Mac, with a 3.12.8 version and I ran that script:
/bin/python3 -i /Users/yves/Desktop/Bugs/gh-132898.py and in REPL, I call main(), I have got errors:

Ouptput 
multiprocessing.get_start_method() = 'spawn' # I print the current `start_method` and keep only two numbers to check
112272535095293 is prime: True
1099726899285419 is prime: False
>>> main()
Process SpawnProcess-4:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/process.py", line 252, in _process_worker
    call_item = call_queue.get(block=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'is_prime' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
Process SpawnProcess-3:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/process.py", line 252, in _process_worker
    call_item = call_queue.get(block=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'is_prime' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/yves/Desktop/Bugs/gh-132898.py", line 31, in main
    for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/process.py", line 636, in _chain_from_iterable_of_lists
    for element in iterable:
                   ^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while

if I changed the start_method to forkserver, I have got the same errors.

And finally if the start method is fork everything is fine.
/usr/local/bin/python3 -i /Users/yves/Desktop/Bugs/gh-132898.py

multiprocessing.get_start_method() = 'fork'
112272535095293 is prime: True
1099726899285419 is prime: False
>>> main()
112272535095293 is prime: True
1099726899285419 is prime: False
>>>

IMO, this issue is also causing by start_method but I don't figure out how REPL is involved.
And a last remark, if you replace the is_prime function with math.sqrt, the script succeeds in any start_method case .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.14 new features, bugs and security fixes stdlib Python modules in the Lib dir topic-multiprocessing type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@YvesDup @picnixz @StanFromIreland @wenming21