Skip to content

Building from git fails: Error compiling Cython file: Pythonic division not allowed without gil #6719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
gerritholl opened this issue Nov 25, 2015 · 9 comments
Closed

Building from git fails: Error compiling Cython file: Pythonic division not allowed without gil #6719

gerritholl opened this issue Nov 25, 2015 · 9 comments
Labels
component: build

Comments

@gerritholl
Copy link
Contributor

Building numpy from git fails when Cythonizing numpy/random/mtrand/mtrand.pyx, due to an error Pythonic division not allowed without gil, consider using cython.cdivision(True). This is Python 3.5.0, Cython 0.23.4, with numpy git revision d94043f0fb93985fd4302eae9fcdca2d79239908 on branch master. The same problem occurs on branch maintenance/1.10.x with commit 76db1ab4edef69e310426dccf95c15212b1ba4a6, and on branch maintenance/1.9.x with commit edb902cdc6573553afcf11047ecdfb447e444322. However, a regular pip install numpy (which installs 1.10.1), works (because releases include compiled cython files, but version control does not). The failures occur as follows:

$ python3.5 setup.py build_ext                                                                       
Running from numpy source directory.
Cythonizing sources
Processing numpy/random/mtrand/mtrand.pyx

Error compiling Cython file:
------------------------------------------------------------
...
            i = 0
            while i < sz:
                Sum = 1.0
                dn = n
                for j from 0 <= j < d-1:
                    mnix[i+j] = rk_binomial(self.internal_state, dn, pix[j]/Sum)
                                                                          ^
------------------------------------------------------------

mtrand.pyx:4484:75: Pythonic division not allowed without gil, consider using cython.cdivision(True)
Traceback (most recent call last):
  File "/tmp/numpy/tools/cythonize.py", line 199, in <module>
    main()
  File "/tmp/numpy/tools/cythonize.py", line 195, in main
    find_process_files(root_dir)
  File "/tmp/numpy/tools/cythonize.py", line 187, in find_process_files
    process(cur_dir, fromfile, tofile, function, hash_db)
  File "/tmp/numpy/tools/cythonize.py", line 161, in process
    processor_function(fromfile, tofile)
  File "/tmp/numpy/tools/cythonize.py", line 72, in process_pyx
    raise Exception('Cython failed')
Exception: Cython failed
Traceback (most recent call last):
  File "setup.py", line 263, in <module>
    setup_package()
  File "setup.py", line 251, in setup_package
    generate_cython()
  File "setup.py", line 199, in generate_cython
    raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

What's going wrong?
@gerritholl gerritholl changed the title Building fails: Error compiling Cython file: Pythonic division not allowed without gil Building from git fails: Error compiling Cython file: Pythonic division not allowed without gil Nov 25, 2015
gerritholl added a commit to gerritholl/numpy that referenced this issue Nov 25, 2015
@gerritholl
BUG: Fix/workaround for numpy#6719
418ef99 
numpy/random/mtrand/mtrand.pyx contains a line where cython fails to
compile, complaining “Pythonic division not allowed without gil”.  By
passing '--directive cdivision=true' to cython, it can compile again.
@gerritholl
Copy link
Contributor Author

I added pull request #6720. I don't know if this is a workaround that fails to address the problem, or whether this is an actual fix, because I am not sure what the underlying problem is and why it hasn't been reported before — if I run into this problem, shouldn't many others too?

@charris
Copy link
Member

charris commented Nov 25, 2015

Probably Cython 0.23.2, I don't see a problem here with Cython 0.22. Running that bit of code with nogil is also in numpy 1.9, so it isn't new.

@charris
Copy link
Member

charris commented Nov 25, 2015

Note that two C floats are involved in the division, so the python version of / should not matter. I think this is a cython glitch.

@gerritholl
Copy link
Contributor Author

Even if it's a bug in cython, wouldn'd numpy still want a workaround? I tried with both 0.23.2 and 0.23.4 so if it is a Cython bug, it's not a bug occurring in just a single revision.

@gerritholl
Copy link
Contributor Author

See pull request #6721

gerritholl added a commit to gerritholl/numpy that referenced this issue Nov 25, 2015
@gerritholl
BUG: Fix for numpy#6719
4e276ac 
numpy/random/mtrand/mtrand.pyx contains a line where cython fails to
compile, complaining “Pythonic division not allowed without gil”.  By
running this code segment under cdivision(True), this problem is avoided.
charris added a commit that referenced this issue Nov 25, 2015
@charris
Merge pull request #6721 from gerritholl/circumvent-cython-problem-lo…
e711c95 
…cally

BUG: Fix for #6719
@charris
Copy link
Member

charris commented Nov 25, 2015

Closed by #6721.

@charris charris closed this as completed Nov 25, 2015
@njsmith
Copy link
Member

njsmith commented Nov 25, 2015

Please do file a bug with cython as well.

@robertwb
Copy link

I tried::

cdef double bad_call(int j, double* pix, double Sum):
    with nogil:
        res = pix[j] / Sum
    return res

With the latest Cython head, with Python 2.7 and 3.4, with and without the -3 option, and was not able to reproduce this. Is there something non-standard in your setup?

@gerritholl
Copy link
Contributor Author

@njsmith I sent an e-mail to the cython-users mailing-list. I don't feel quite confident enough to formulate what exactly the cython bug is, as I've never used cython myself.

@robertwb As far as I'm aware, I have not personally set anything non-standard. The machine configuration is different from a typical workstation, though — I'm building on a Red Hat Enterprise Linux Server release 6.7, cython and co installed in my home directory on a panfs filesystem, trying to build on a tmpfs filesystem. I don't think the filesystem type should matter but it's the only unusual aspect I'm aware of. Could you think of some other non-standard things I should look at specifically?

charris pushed a commit to charris/numpy that referenced this issue Nov 26, 2015
@gerritholl @charris
BUG: Fix for numpy#6719
3f743d2 
numpy/random/mtrand/mtrand.pyx contains a line where cython fails to
compile, complaining “Pythonic division not allowed without gil”.  By
running this code segment under cdivision(True), this problem is avoided.
@charris charris mentioned this issue Nov 26, 2015
Merged
colbych added a commit to colbych/numpy that referenced this issue Dec 6, 2015
@colbych
Merge branch 'master' of git://github.com/numpy/numpy into first_patch
441f1a5 
* 'master' of git://github.com/numpy/numpy: (24 commits)
  BENCH: allow benchmark suite to run on Python 3
  TST: test f2py, fallback on f2py2.7 etc., fixes numpy#6718
  BUG: link cblas library if cblas is detected
  BUG/TST: Fix for numpy#6724, make numpy.ma.mvoid consistent with numpy.void
  BUG/TST: Fix numpy#6760 by correctly describing mask on nested subdtypes
  BUG: resizing empty array with complex dtype failed
  DOC: Add changelog for numpy#6734 and numpy#6748.
  Use integer division to avoid casting to int.
  Allow to change the maximum width with a class variable.
  Add some tests for mask creation with mask=True or False.
  Test that the mask dtype if MaskType before using np.zeros/ones
  BUG/TST: Fix for numpy#6729
  ENH: Avoid memory peak and useless computations when printing a MaskedArray.
  ENH: Avoid memory peak when creating a MaskedArray with mask=True/False (numpy#6732).
  BUG: Readd fallback CBLAS detection on linux.
  TST: Fix travis-ci test for numpy wheels.
  MAINT: Localize variables only used with relaxed stride checking.
  BUG: Fix for numpy#6719
  MAINT: enable Werror=vla in travis
  BUG: Include relevant files from numpy/linalg/lapack_lite in sdist.
  ...
jaimefrio pushed a commit to jaimefrio/numpy that referenced this issue Mar 22, 2016
@gerritholl @jaimefrio
BUG: Fix for numpy#6719
3cbc9d4 
numpy/random/mtrand/mtrand.pyx contains a line where cython fails to
compile, complaining “Pythonic division not allowed without gil”.  By
running this code segment under cdivision(True), this problem is avoided.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component: build
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@charris @rgommers @robertwb @gerritholl @njsmith