Skip to content

ENH: add support for array-likes in polybase __call__ #18639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 53 commits into from
Closed

ENH: add support for array-likes in polybase __call__ #18639

wants to merge 53 commits into from

Conversation

cval26
Copy link
Contributor

@cval26 cval26 commented Mar 17, 2021

ENH: add suport for array-likes in polybase __call__

Closes #17949.

Adds an asarray() call in the polybase __call__ method so that evaluation works with array-likes too. See #17949 for the initial reproduced code example; after the above change, evaluation works as expected:

import numpy as np
>>> np.polynomial.Polynomial([1, 2])(np.array([3, 4]))
array([7., 9.])
>>> np.polynomial.Polynomial([1, 2])([3, 4])
array([7., 9.])

@charris charris changed the title ENH: add suport for array-likes in polybase __call__ ENH: add support for array-likes in polybase __call__ Mar 17, 2021
@charris
Copy link
Member

charris commented Mar 21, 2021

Needs rebase and release note.

aerikpawson and others added 25 commits March 21, 2021 19:25
Specifying color in RGB in the guidelines for the new NumPy logo. Fixes #gh-398 numpy/numpy.org#398
Removing the unofficial names of the logo colors.
This reduces the number of cases when floating point precision makes the argument to `log` become `0`
Use log1p(-x) instead of log(1 - x)
Seperate legacy version from current

closes #17020
Improve the exception when low is 0 in case the single input
form was used.

closes #14333
I found these convenient to have on hand while reading
When creating the `bad_commands` dictionary it is useless to declare commands that are already set with a default message in the 329th line's for loop
This is a fast argument parser (an original version also supported
dictionary unpacking (for args, kwargs call style) which supports
the new FASTCALL and VECTORCALL convention of CPython. Fastcall
is supported by all Python versions we support.

This allows todrastically reduces the overhead of methods when keyword
arguments are passed.
Array methods are the easiest target for the new parser. They
do not require any larger improvements and most functions in
multiarray are wrapped. Similarly the printing functions
are not dispatched through `__array_function__` and thus have the
most to gain.
While introducing the buffer fixed the in-place problem years ago,
running valgrind (and masked arrays) pointed out to me that without
the additional `...` NumPy will unpack and repack objects leading
to slightly incorrect results.

MAINT: Warn about shuffle bug instead of fixing it in old random API
This changes all casting loops to get passed in a new `context`
which holds some data such as the descriptors (and methods).
For the most part, this is unnecessary information, but it
allows us to pass additional information to ufuncs while also
aligning ufunc and casting signatures (and definition).

To achieve this, we have to fill and store this context
everywhere, which leads to some rework.
That means that this PR changes some of the chained loops
to use fewer specialized loops.

There should be no typical speed effect for larger arrays
aside possibly a small one for rarer cases such as byteswapped
or unaligned arrays (although we currently consider strings
unaligned much too often).

However, there will be a small increase in overheads, timing
it, it appears to be <10% in the most extreme cases.
While not neglible there are bigger optimizations possible
(even moving the buffer-info data to the array struct was
a bigger speedup in most cases!).

Below are individual (squashed) commit while working on this,
since the change had to be mostly complete to compile/test
its fairly long (leaving it just for fun)

Individual commit
-----------------

Further changes

Fixup the main multi-step cast functions

Restructure field transfers (partially?)

continuation

WIP: This is tricky, need to redo some stuff (this is the start of that redo)

WIP: more small changes

And more work...

Clean out/simplify legacy multi-step casts (mainly datetime)

Fixup strided one to n field transfer (and decref partially)

Fixup n-to-n cast for new methods

Fix subarray casts for the new calling convention

Finish fixing up the structured transfers and decref

start with cleaning up masked casts

BUG: Fixup struct access typo

Fixup any-to-object

Finish all/most of masked casting, this brings me to debugging...

Fixup object copy and move functions

Code now compiles without errors (of course segfaults immediately :))

Finish fixup of array_assign_*

Pass pointer to cast_info_xfree (to be able to NULL func and error check)

Also silence a few warnings

Bug: Make sure datetime casts have copy/copyswap only path

This is necessary to ensure the recursive calls don't recurse
forever :).

Small fixes, mostly make sure we use PyMem_* for all allocations/frees

BUG: Fix memset in multistep cast

Seems that code was little enough tested, that most test runs would
simply not notice the issue (i.e. requires debug mode usually probably).
Now it is used a lot and things came down hard.

Bug: Fix stride order typo affecting strided masked indexing

BUG: Finish npy-iter copy fixup (apparently it was not...)

BUG: Fix scalar assignment mixup

Fix bug in field bookkeeping and rename decsrcref to decref

Its called "decref" everywhere, the "src" is just confusing
especially since now we don't have the `func(dst, src)` calling
convention anymore.

Final fixes and cleanup of comments

e.g. removed some comments alluring to a future that is unlikely
to ever come...

DOC: Change casting error message to be closer to other messages

Fixup rebase: masked transfer function initialize got lost

Fixup: remove print added for debugging into recfunctions
These is probably fairly common, so optimize the lookup in this
extremely simple case.

More optimizations should probably done eventually, e.g. we could
add a fast-list for all builtin ones, or have an "intern" mechanism
so that casts for any interned dtype will be stored in a list, with
the assumption that an interned dtype can e.g. never be deleted
and that there should not be too many of them.
This could maybe be done a bit nicer, for now a small (local) helper
seemed convenient.  This is only needed here.
There were some paths that manipulated the unused data. This was
fixed in the big refactor (or earlier), I expect there were some
fast paths for equal sized dtypes that kicked in before
As noted by Matti and Ross, these comments were incorrect.

Co-authored-by: Matti Picus <matti.picus@gmail.com>
Co-authored-by: Ross Barnowski <rossbar@berkeley.edu>
Not the prettiest tests, but should ensure that casts involving
byteorder changes within datetime/timedelta and between datetimes
or timedeltas and strings are handled correctly.
It seemst he unicode didn't actually help coverage, but here we go.
This is utterly confusing, but adds coverage to another uncovered
pass.

I honestly do not really understand right now, why "writemasked"
and "buffered" combination can bypass the buffering here.
I guess there some special logic about that in the `where=` ufunc
code that uses the masked versions here?
* Added chain exception in _bits_of func

* Added chain exception

* Added chain exception in unixccompiler.py

* Added chain exception in config.py

* Added chain exception in fcompiler __init__.py

* Added chain exception in compaq.py

* Added chain exception in format.py

* Updated raise chain exception

* STY: Break long line.

Co-authored-by: Charles Harris <charlesr.harris@gmail.com>
aitikgupta and others added 26 commits March 21, 2021 19:27
Co-authored-by: Eric Wieser <wieser.eric@gmail.com>
Co-authored-by: Eric Wieser <wieser.eric@gmail.com>
* Added chain exception in _pocketfft.py

* Added chain exception in format.py

* Added chain exception in make_lite.py

* Added chain exception in mrecords.py

* added from e for exceptions

* Minor update for _read_array_header exception msg

* Removed \n from excp msg and e from msg format.

Co-authored-by: Eric Wieser <wieser.eric@gmail.com>

* Update numpy/linalg/lapack_lite/make_lite.py

Co-authored-by: Eric Wieser <wieser.eric@gmail.com>

Co-authored-by: Eric Wieser <wieser.eric@gmail.com>
Co-authored-by: Charles Harris <charlesr.harris@gmail.com>
As per NEP 29, Numpy < 1.15 is dead, so there is no need to mention it in the README.
Add missing type to cdef statement
Small spelling fix
* BUG: Fix small valgrind-found issues

This should be backportable. There was at least one that I could
not reproduce when running the tests again. And the new random-shuffle
tests give false-positives (which is just slightly annoying, considering
that we are very close to almost only "longdouble" related
false-positives)

* BUG: Add missing decref in user-dtype fallback paths

The missing decref here only leaks references and can never leak
actual memory fortunately.

* MAINT,TST: Simplify the "refcount logic" in the dispatch tests again

Using SETREF can be nice, but was just overcomplicating thing here...
The array creation functions have the most to gain:

1. np.asarray is 4 times faster and commonly used.
2. Other functions are wrapped using __array_function__ in Python
   making it more difficult

This commit (unfortunatly) has to do a few things:

* Modify __array_function__ C-side dispatching to accomodate
  the fastcall argument convention.
* Move asarray, etc. to C after removing all "fast paths" from
  np.array (simplifying the code)
* Fixup imports, since asarray was imported directly in a few places
* Replace some places where `np.array` was probably used for speed
  instead of np.asarray or similar. (or by accident in 1 or 2 places)
The argument name for the python side defined function was always
`a` and not `object` so fix it and add a test. Also add a test to
ensure the error paths are taken for all of the function.

The missing NULL returns cannot be tested easily (but thanks to
code coverage its nice to look at the uncovered stuff closer...)
I had forgotten that the docs for these functions where updated
when rebasing.
On the C-side binops only guarantee that one of the operands
is an instance of `PyArrayObject *`, but we do not know which
one.  Typing the first as `PyArrayObject *` is just misleading
for almost no actual payoff
Co-authored-by: Matti Picus <matti.picus@gmail.com>
This can currently only be passed positionally, and this is
undocumented.
Including this directory is painful for cross-compiling, see
gh-14980 and gh-13280. Removing this directory fixes the following
build failure:
```
gcc: numpy/core/src/common/numpyos.c
In file included from numpy/core/src/common/numpyos.c:23:
/usr/include/xlocale.h:27:16: error: redefinition of ‘struct __locale_struct’
   27 | typedef struct __locale_struct
```
This error also shows up in various other build issues outside of the
NumPy issue tracker.

Compilers normally always include this path, so this shouldn't break
anything. The default include paths for the compiler can be checked,
e.g. for gcc with `cpp -v`. That will typically have /usr/include last.

In case this breaks something for a nonstandard compiler, that can be
worked around via a site.cfg file in the root of the repo (or
equivalently, `~/numpy-site.cfg`)  containing:
```
[DEFAULT]
include_dirs = /usr/include
```

The same principle should apply to `/usr/lib`. I will make that change
in a separate commit, because the failure mode for that will be
different (and I'm not running into it right now).
- 3D Axes are created via add_subplot(projection='3d')
- There is now a `stairs()` function that's specifically designed for
  showing histogram curves
- Labels should be passed as keyword arguments to the plot functions
  instead of to `legend()`, which reduces the risk of mixing them up.
- ensure equal axis scaling in the meshgrid example
@cval26 cval26 closed this Mar 21, 2021
@cval26 cval26 deleted the polybase-call branch March 21, 2021 23:39
@cval26
Copy link
Contributor Author

cval26 commented Mar 21, 2021

@charris I had a problem with the rebase; closed and opened new PR #18662

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Polynomial can't be evaluated on list of numbers only ndarray of numbers