Ping @rafaelcfsousa, could you have a look, since it was likely your PR that introduced this?

@seberg: Yes, I am already checking the issue.

Ok, I already know what exactly is causing this bug.

The comparison kernels implemented with AVX and AVX512 use the universal intrinsic npyv_cmpneq_f64 (see below) to compute np.not_equal for dtype=double.

The universal intrinsic npyv_cmpneq_f64 for both SIMD extensions listed above uses the flag _CMP_NEQ_OQ to indicate how the comparison with NaN has to be done (more info here: https://stackoverflow.com/questions/16988199/how-to-choose-avx-compare-predicate-variants).

For _CMP_NEQ_OQ (Ordered comparisons returns false for NaN operands):

• nan != nan --> false
• nan != 0 --> false

For _CMP_NEQ_UQ (Unordered comparison returns true for NaN operands):

• nan != nan --> true
• nan != 0 --> true

I assume this doesn't need a backport?

In my point of view, we could follow one of the 4 options below to have this issue resolved:

1. Use _CMP_NEQ_UQ instead of _CMP_NEQ_OQ (but I do not know yet the impact of doing that)

2. Disable the optimization for AVX[2,512] so that SSE2 is used instead

3. Move the dtypes f32 and f64 to a new dispatchable source file (loops_comparison_fp.dispatch.c.src) and enable on it only SSE (and the other architectures)

4. Use scalar execution for AVX[2,512] when dtype=f32 or f64

Using _CMP_NEQ_OQ seems right to me on first sight, unless this is used in places where the unorderd version is needed (in which case, I guess we may need both universal intrinsics?)

EDIT: Whoops, copied the wrong intrinsic there...

@seberg and @charris :

I think we should use _CMP_NEQ_UQ instead of _CMP_NEQ_OQ.

>>> import numpy as np
>>> np.nan != np.nan
True
>>> np.nan != 0
True

The result when I compare anything with NaN is True.

Btw, npyv_cmpneq_f32 and npyv_cmpneq_f64 are only used by loops_comparison.dispatch.c.src.

Sorry, yes. I got the wrong intrinsic, I meant: I do think we should swap out the intrinsic/flag used and that seems like it should be the right solution. Are you planning on making a PR?

Yes, I will submit a PR. Give me ~1h and I will submit it (I will test the other comparison function as well).

FWIW: Locally, I just tried switching _CMP_NEQ_OQ to _CMP_NEQ_UQ for npyv_cmpneq_f32 and npyv_cmpneq_f64. The existing tests still passed, and the example that I gave above worked correctly for float32 and float64. @rafaelcfsousa, thanks for tracking down the issue so quickly!