only have the .c.src in the setup.py, not both.

couldn't this just be a regular ufunc?

I have the same question as @juliantaylor, what is the use case for making this a gufunc? If multiple axis are supported I think the same could be accomplished with a ufunc. There would be some work involved as this isn't the standard reduce operation. We should probably give some thought to what general class of operations all_* functions fall under.

@juliantaylor, @charris - the difference is the reduction in dimensionality here. Are there ways other than reduce that do this for a ufunc? (reduce would not apply here, of course, as it only takes a single input array, and the whole point is to avoid calculating the full boolean array).

latest commit is probably a hack to get it to build on my machine, but fails for some builds. I tried @juliantaylor advice but couldn't get it building on my machine, probably do to my unfamiliarity with numpy builds. At least it allows me to start working on tests and other aspects.

It seems that a creating similar functionality with a regular ufunc would require a large rewrite of the ufunc machinery. Numpy would need to add something like reduce for non-binary ufuncs and also have a short circuit mechanism, neither seems easy IMHO. It seems like generalized ufuncs were created for cases like this.

I am unsure whether this functionality is welcomed or not. If not, I would prefer to know so that I don't spend any more time on it. Should it be explicitly asked in discussion?

It is a bit from the peanut gallery, but I would definitely use it!

Looks like you may need to add something to MANIFEST.in for the sdist. Not sure why...

It seems like generalized ufuncs were created for cases like this.

I agree -- this feels like an excellent fit for generalized ufuncs.

I am unsure whether this functionality is welcomed or not. If not, I would prefer to know so that I don't spend any more time on it. Should it be explicitly asked in discussion?

I am also a fan and would support including this in NumPy. These types of comparisons are done all the time, and for some use cases this could be a significant improvement.

@juliantaylor Unused function compiler warning.

yes it should just use a header file instead of including a C file.

Can you please explain what should go in the header and why? I tried a few things but only got segfaults. I apologize for the newbie questions. Thanks.

Maybe a bit silly, but I am still wondering a bit about something like np.equal.all(). I.e. some ufunc mechanism which uses the typical ufunc loops instead of specialized loops (and probably would support loops returning bools, though that does not matter much). On the up-side it would allow much more generality. On the downside, you would have to live in a buffered world (even for the trivial cases, you will have to do do chunking logic) with chunking to get the "early stop" behaviour (i.e. you do not have a perfect stop as soon as one false/true occurs, but only "stop soon after". Which probably does not matter much speed wise, though may give some wrong warnings (or even errors for objects).

@juliantaylor Thanks for the feedback. I fixed the easy ones. I don't know what you mean by "templated functions" or the docstring "templating script". Can you point me to examples?

the /*begin repeat comments are our form of templates, it can be used on everything in a file, so it should also be usable to create all the very similar docstrings from one template.

Grep is your friend.

/**begin repeat
 * #TYPE = BYTE, UBYTE, SHORT, USHORT, INT, UINT,
 *         LONG, ULONG, LONGLONG, ULONGLONG#
 * #type = npy_byte, npy_ubyte, npy_short, npy_ushort, npy_int, npy_uint,
 *         npy_long, npy_ulong, npy_longlong, npy_ulonglong#
 * #ftype = npy_float, npy_float, npy_float, npy_float, npy_double, npy_double,
 *          npy_double, npy_double, npy_double, npy_double#
 * #SIGNED = 1, 0, 1, 0, 1, 0, 1, 0, 1, 0#
 */

@juliantaylor I agree vectorization would be nice. This function is indeed slower for cache sized arrays where the entire array must be checked. On my personal machine is a little old (i3 M370, no AVX):

In [3]: x = np.arange(2**16, dtype=float)
In [5]: y = x.copy()
In [8]: %timeit (x==y).all()
10000 loops, best of 3: 81.8 µs per loop

In [9]: %timeit np.all_equal(x,y)
10000 loops, best of 3: 116 µs per loop

The new function is still faster for huge arrays, tiny arrays, or cases where an early element is not equal. However it would be nice if it were faster for all problems. I would like to learn more about vectorization, this seems like my chance!

vectorization is not necessary for now, it can be added later too.

Here is a toy function which vectorizes nicely:

#define ARR_LENGTH 4096
typedef float afloat __attribute__ ((__aligned__(32)));
int curly(afloat *a, afloat *b){
    unsigned int i;
    unsigned int result = 0;
    for (i=0; i<ARR_LENGTH; i++){
        result += (*a) == (*b);
        a++;
        b++;
    }
    return result == i;
}

Compiled with gcc 6.3 -O3 -march=haswell yield nice looking avx loop

.L2:
        vmovaps (%rsi,%rax), %ymm0
        vcmpeqps        (%rdi,%rax), %ymm0, %ymm0
        addq    $32, %rax
        vpsubd  %ymm0, %ymm1, %ymm1
        cmpq    $16384, %rax
        jne     .L2

☔ The latest upstream changes (presumably #8043) made this pull request unmergeable. Please resolve the merge conflicts.

I still need to do housekeeping for docstrings and release notes. However I think this is getting close.

This is my first serious attempt at vectorization. I'm relying on the compiler to automagically vectorize the code. It seems to be pretty quick. There is probably room for improvement though. Here is a crude benchmark on my old personal machine. I would like to know how it does with AVX. Also how much should be vectorized? Right now its not much, just all_equal for int, float, and double.

In [5]: x = np.ones(2**18, dtype=np.float32);y=x.copy()
In [6]: %timeit (x==y).all()
1000 loops, best of 3: 203 µs per loop
In [7]: %timeit np.all_equal(x, y)
10000 loops, best of 3: 145 µs per loop

@charris previously commented:

Docstrings should be handled as ufunc docstrings are, not in C code. See numpy/core/code_generators/ufunc_docstrings.py.

AFAIK in order to have docstrings like that you also need to use numpy/core/code_generators/generate_ufunc_api.py. I have only looked at that very briefly, but its not clear to me that mixing a python generator script with C code helps all that much here. The .src templating scripts work really well here IMHO. The only real benefit I see in switching is consistency with what already exists.

1. Am I correct that you need to use genereate_ufunc_api to use ufunc_docstrings.py?
2. Should I make that switch?

Maybe a bit silly, but I am still wondering a bit about something like np.equal.all()

+1 on this. This would then also provide np.logical_and.(any|all) and np.logical_or.(any|all), although these don't strike me as immediately useful.

With a bit of type coercion, you could allow np.add.any(a, b) == np.any(a + b) == np.any(a + b != 0), although that's a little magic

Maybe a bit silly, but I am still wondering a bit about something like np.equal.all()

Would be visually nice, but in the context of ufuncs it remains odd to me. The functions here rather logically fit the gufunc mold, as it really is two ufunc's chained together (logical_and.reduce(equal(a, b))), but then nicely done in an inner loop. Furthermore, the short-circuiting is very particular to the logical functions, not general to all.

Now the first point does hint at a broader question, whether it would be possible to chain more generally (e.g., add.reduce(power(a, 2)), or the ability to multiply an array by a constant before applying a ufunc, as in, sin(multiply(a, b)) or add(a, multiply(b, c)) (i.e., stuff done by numexpr...)

Maybe this PR is a first step towards what I just said.

In that case, it's interesting to look at some timings:

[1]: z = np.eye(10000, dtype='i')
[2]: x = np.zeros((10000,10000), dtype='i')

# test for for every possible index of first inequality
[3]: %timeit [np.any_not_equal(zi, xi) for zi,xi in zip(z,x)]
30.8 ms ± 2.47 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
[4]: %timeit [np.logical_or.reduce(zi != xi) for zi,xi in zip(z,x)]
99.2 ms ± 5.99 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# test when the first inequality is at index 10
[5]: zi, xi = z[10], x[10]
[6]: %timeit np.any_not_equal(zi, xi)
1.36 µs ± 27.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
[7]: %timeit np.logical_or.reduce(zi != xi)
8.33 µs ± 295 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

So this PR does appear to give a speedup of about 3x on average, or much more if we short-circuit early.

Maybe the way forward is this: Merge this PR, and then in future PRs specialize it to 1) Add loops comparing to a scalar (i),()->() and loops comparing to 0 (i)->(), and 2) Adapt this same code to compute first/count.

If @mattharrigan is still around, I would be interested in following this through.

I'd prefer to go straight to ufunc.first and ufunc.count rather than adding more api surface to np. first by merging this.

This PR currently adds these ufuncs at toplevel, np.all_not_equal and so on. That's what you're talking about, right?

I agree I don't think they should be top level, I think they should be exposed as np.not_equal.all.

@mattharrigan it might ultimately be nice to have all of them: np.equal.all, np.equal.any, np.equal.first, and np.equal.count.

So maybe instead of trying to do everything at once we should get any and all in since they are already here.

Technical question: How can we add these two new gufuncs as attributes on the ufunc objects (like np.equal.all)? It's not trivial.

One possibility is to add an any and all method to all ufuncs (in ufunc_methods), and make them raise an error for the ufuncs where this operation is inapplicable, which is most of them. That's how we already do it with reduce/outer/at. To complete this PR that way we would need to write some setup code in ufunc_object.c to detect and reject all the invalid ufuncs, and we would possibly need another entry in the PyUFuncObject struct to store the location of the corresponding gufunc loops. It seems a bit messy and clutters the attributes of all ufuncs.

Another possibility is to use monkeypatching to only give the valid ufuncs the new methods in python. One obstacle here is that ufuncs do not currently support monkeypatching (or inheritance). But I figured out the minimal changes to this PR to get monkeypatched any and all to work, see the code here (link). I had to take over an unused pointer in the PyUFuncObject struct to do this without affecting the struct layout. It's not perfect and a bit ugly, but it works.

Does anyone find one of these options (or a third one) preferable? Is it acceptable to take over the unused pointer?

Or, should we put off this question to another PR, and merge this one? In that case I suggest we add a leading underscore to these gufuncs (so all_equal becomes _all_equal), thus hiding them for now until we decide what to do. We then only need conflict resolution and an update to the test.

@ahaldane - I quite like the monkey-patching except that it does seem to lead to illogical behaviour for __array_ufunc__: where for reduce, etc., one gets the regular ufunc and with method="reduce", here for a similar attribute one would get passed a gufunc.

In that respect, it might be better to define them generally, although as you say it is quite annoying that they then would appear among the methods for all ufuncs. Ideally, we'd have inheritance and define logical ufuncs that do have all and any (and similarly have the 2-argument ufuncs that can deal with reduce inherit from a base class that does not have that attribute...).

p.s. Not sure that I like merging anything unless we're exposing it: a private ufunc either means that it will see no use, or that people use it and it effectively becomes public anyway.

@mhvk I hadn't realized the __array_ufunc__ problem. That may also be a problem the day we convert the reduce methods to their own gufuncs, which I think is the plan. Perhaps we can get around it by giving all these gufuncs an attribute pointing to their "base/scalar" ufunc, and arrange it so they pass the scalar ufunc to __array_ufunc__. Or maybe we can one day deprecate the "method" keyword once we only have only ufuncs/gufuncs.

As for inheritance, I also thought about it but I think it will be messier than monkey-patching. The problem is that some ufuncs may belong to multiple types, eg both to "reduce-like" and "logical", meaning we may need to create complex mixtures of mixins for each ufunc. Each ufunc would be a singleton for its subclassed type, and the scalar ufuncs will no longer simply have numpy.ufunc type. That seemed uglier than monkey-patching.

As for merging, you're probably right. Probably the best course of action is to make a separate PR to decide how to add new methods to ufuncs, and once that is merged fix up this one.

I very much like np.equal.all and would like to help make it happen. I also don't like that some ufuncs have methods defined which don't actually work (np.sin.reduce for instance). As another option, could each ufunc be a class and simply define 0 or more methods in C, as opposed to each ufunc being an object of the same class? Also I'm curious about the history and why all ufuncs are objects of a single class, and what obstacles might be to changing it now.

There seems to be consensus that something like this would be useful.

@mattharrigan Are you interested in continuing this? It needs some rework at this point, but it seems folks would like having it.

I'm going to close this as inactive. Adding this functionality seem popular, so marking as good idea.