[WIP] backport from master to the 1.7.x release branch #439
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(Broken by PR numpy#350.) Should be applied to maintenance/1.7.x as well.
Mutating a bytes object is theoretically unsafe, but doesn't cause any problem in any existing version of CPython.
Previously NumPy did not compile in Python 3.3 due to the changes in PEP 393.
This patch simply calls PyUnicode_FromKindAndData() from the new Python 3.3 API
and possibly swaps the data before calling it if needed. The data in NumPy is always UCS4 and the PyUnicode_FromKindAndData() internally converts it to UCS1, UCS2 or UCS4 depending on the maximum code point.
The following tests now fail, because they produce invalid unicode in the
process (will be fixed in the next patch):
======================================================================
ERROR: Check byteorder of 0-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 277, in test_values0D
self.assertTrue(ua[()] != ua2[()])
SystemError: invalid maximum character passed to PyUnicode_New
======================================================================
ERROR: Check byteorder of multi-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 297, in test_valuesMD
self.assertTrue(ua[0,0,0] != ua2[0,0,0])
SystemError: invalid maximum character passed to PyUnicode_New
======================================================================
ERROR: Check byteorder of single-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 286, in test_valuesSD
self.assertTrue(ua[0] != ua2[0])
SystemError: invalid maximum character passed to PyUnicode_New
======================================================================
ERROR: Check byteorder of 0-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 277, in test_values0D
self.assertTrue(ua[()] != ua2[()])
SystemError: invalid maximum character passed to PyUnicode_New
======================================================================
ERROR: Check byteorder of multi-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 297, in test_valuesMD
self.assertTrue(ua[0,0,0] != ua2[0,0,0])
SystemError: invalid maximum character passed to PyUnicode_New
======================================================================
ERROR: Check byteorder of single-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 286, in test_valuesSD
self.assertTrue(ua[0] != ua2[0])
SystemError: invalid maximum character passed to PyUnicode_New
The tests are testing byte order for unicode, so we can only use such unicode data, so that both versions (swapped and unswapped) are valid unicode.
Temporary array was not being freed after use.
This function handles the swapping automatically and it returns a unicode object in one of: UCS1, UCS2 or UCS4 internal Python format.
It comes from the Python compiler package, which isn't available on Python 3.x. We already handle that issue by instead importing the ast module.
Fix returned copy so that copy of view with offsets copies only fields in view, not all the fields from original array. Also add unit tests to make sure this doesn't break when copy is fully deprecated in favor of returning a view.
This patch enforces a strict dichotomy for the variables 'indescr' and 'newdescr', so they are either NULL, or they own a reference. Following the consequences of this allowed the reference error to be tracked down.
The bug was fixed by the previous patch.
|
I would just merge them as you confirm each one is correct, instead of trying to review a giant pile together... Anyway skimming the list above: 5a471b5 is a new optimization, not a bugfix, so technically shouldn't be merged into the stable branch, but it's small enough that it probably won't matter. 56596a0fa04c7474967747d21a9df61a95edca62 was an error and immediately reverted in master, so it shouldn't be here. |
This should fix the problems with numpy.insert(), where the input values were not checked for all scalar types and where values did not get inserted properly, but got duplicated by default.
The problem was that in 32bit Ubuntu 12.04, one gets the following: > /home/njs/numpy/.tox/py27/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py(363)test_pareto() -> np.testing.assert_array_almost_equal(actual, desired, decimal=15) (Pdb) actual[1, 0] 52828779.702948704 (Pdb) desired[1, 0] 52828779.702948518 and the test was comparing the numbers to 1e-14, which obviously failed. Fixes numpy#424.
Travis recently upgraded to virtualenv 1.8, which has dropped support for Python 2.4. So, in our Python 2.4 setup script, we need to explicitly fetch and use virtualenv 1.7. Likewise for pip 1.1. File imported from the already-fixed version for patsy: https://github.com/pydata/patsy/blob/0316d2901f4195db06e8091c15f37d9fe4ad09de/.travis-make-py24-virtualenv.sh
|
Thanks! I've removed the 56596a0 as well as your commit fixing it (which added the keyword arguments, but that can just stay in master). As far as Stefan's commit 5a471b5, I tried to remove it, but then his other commit "Retain backward compatibility. Enforce C order." does not apply cleanly. So I would have to manually fix the conflicts, so I think it's better to just keep both patches in, so that it's clear where they come from. |
|
P.S. I want to run Travis tests on this, as well as keep it here for other people to see, rather than just pushing it in. |
|
Ok, this looks good. One PR is missing, but I will simply note it at the TODO list and merge it in the next batch. All tests pass and as far as I can see, this looks good. So I am merging this. |
|
I just cherry-picked the #432 manually. So all is good. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I have backported the patches from master. There is a lot of patches, so I still need to carefully go through them and determine whether we need to apply all of them or only a subset.
I will be rebasing this branch, dropping or adding commits until it is polished and ready to merge.
If you see any problems, let me know.