The Ubuntu 22 runner stopped because test-backend-ops crashes with segfault. I've observed this behavior on a raw build before as well so I don't know if it has anything to do with my modifications or if it makes this happen more likely.

Edit: when the app crashes with segfault it is always during the exit after the tests were executed.

It's showing failures in some of the conv_transpose_1d tests, e.g.:

29:   CONV_TRANSPOSE_1D(ne_input=[2173,1,1,1],ne_kernel=[1337,1,1,1],s0=1,p0=0,d0=1): [CONV_TRANSPOSE_1D] NMSE = 0.459058786 > 0.000000100 FAIL
29:   CONV_TRANSPOSE_1D(ne_input=[2173,1,1,1],ne_kernel=[1337,1,1,1],s0=2,p0=0,d0=1): [CONV_TRANSPOSE_1D] NMSE = 0.222932697 > 0.000000100 FAIL
29:   CONV_TRANSPOSE_1D(ne_input=[2173,1,1,1],ne_kernel=[1337,1,1,1],s0=3,p0=0,d0=1): [CONV_TRANSPOSE_1D] NMSE = 0.167964696 > 0.000000100 FAIL

Then it appears to crash on the first test after the conv_transpose_1d tests, which does seem like it's related to this change somehow.

It's showing failures in some of the conv_transpose_1d tests, e.g.:

29:   CONV_TRANSPOSE_1D(ne_input=[2173,1,1,1],ne_kernel=[1337,1,1,1],s0=1,p0=0,d0=1): [CONV_TRANSPOSE_1D] NMSE = 0.459058786 > 0.000000100 FAIL
29:   CONV_TRANSPOSE_1D(ne_input=[2173,1,1,1],ne_kernel=[1337,1,1,1],s0=2,p0=0,d0=1): [CONV_TRANSPOSE_1D] NMSE = 0.222932697 > 0.000000100 FAIL
29:   CONV_TRANSPOSE_1D(ne_input=[2173,1,1,1],ne_kernel=[1337,1,1,1],s0=3,p0=0,d0=1): [CONV_TRANSPOSE_1D] NMSE = 0.167964696 > 0.000000100 FAIL

Then it appears to crash on the first test after the conv_transpose_1d tests, which does seem like it's related to this change somehow.

That's interesting. I could finally reproduce these errors with llvmpipe, but not with the discrete GPU. I need time to investigate what's the root cause.

Wild guess - any chance of uninitialized shared memory?

Wild guess - any chance of uninitialized shared memory?

It can be, but there is a chance that the code is correct but llvmpipe what's the root cause of the failing tests.

I executed parameter sweeps (Cin, K, L) and turns out that if one (or combination of parameters) are large enough, the test fails. Even if I test with Cin=20000, L=64, K=4 the test fails that's suspicious because Cin "does not influence the logic too much". Cin=20000, L=64, K=3 passes.

Argmax also segfaults on llvmpipe on my computer for some reason.

Wild guess - any chance of uninitialized shared memory?

No, I was not aware of the fact that iterations in loops are simply skipped after reaching a certain limit on llvmpipe, so I decided to reduce the maximum test problem size to 1337x13 (input x kernel) in order to to pass the tests on llvmpipee in commit 2813cf4. Llvmpipe starts to trim the loops after ~30k iterations so this number should be safe and also realistic opposed to my previous edge cases. 5120x512 also worked for me with 128 threads resulting in ~20k iterations per thread.

@slaren @0cc4m @jeffbolznv any plans to launch the pipelines to see if this can be merged finally?

Thank you!