Bytes, hehe :). So the thing is the method object is completely opaque right now (which could hold static data). The other options would be to expand context with a static data slot that can be set, I suppose.
Somehow, I didn't think static data was all that important, but folding it into auxdata isn't that much easier of course. But for this type of tihng it is useful (i.e. you don't care about the singleton function call overhead at all).

the method object is completely opaque

But how would I set the method inside, say, init_ufunc?

Also, for these purposes, it really should be extra data that goes with the specific loop, which makes auxdata a bit more logical. Could be part of the spec, I guess (which is effectively what I'm doing by using the name).

Note that in the FFT additions, I similarly made use of auxdata to avoid creating duplicate pieces of code...

So, we could add it to the self as a slot and create accessor (or make it public to the extend needed for that). It is currently opaque (so even if you add it, the user can't get it).

Of course to some degree you could also prefer tagging it on to the ufunc. But than you need to extend the way the ufuncs store loops now (since ufuncs assume evertying is in the ArrMethod object now).

Not sure what is the best approach, but you are right that for this use case, something associated with the ufunc would be good enough, maybe like a "default auxdata" (right now, I see that in npy_default_get_strided_loop, the data just gets set to NULL). Though the loop-specific case is what would be needed for, say, np.exp using a generic inner loop that just calls a given function on each element (where the function likely is specific to the input dtype). I feel an ability to set auxdata as part of adding a given loop would be the most generally useful, i.e., it feels to me easiest would be to add a user_data entry to PyArrayMethod_Spec.

I saw that you had a note in _dtype_api.h that perhaps is relevant:

This refers to filling it if not NULL, but how in the present scheme can auxdata/transferdata/loop_data be anything but NULL?

You can set auxdata just fine, it is done in the get_loop function. And auxdata is in fact what is used internally to pass the static void * into legacy ufuncs.

I.e. you can do:

struct {
   NpyAuxdata base;
   void *func;
} static_auxdata_struct;

/* Do nothing free, it's static */
static void
no_free(NpyAuxData *aux) {}

static_auxdata_struct min_auxdata = {
    .base={
        .free = no_free;
        // Don't bother about clone for now
    }
    func = &string_min;
}

get_loop(...) {
   if (ufunc == ...) {
       out_auxdata = min_auxdata;
   }
}

Which would mean you move the ufunc inspection into get_loop, but that's about it.

That's possible, I guess, but feels wrong; get_loop really should not have to worry about what ufunc it is attached to; at most, it should move some bits of data from its context to auxdata (I guess that is also possible, by defining one's own PyArrayMethod just like is done for the legacy ufuncs...).

Note that making this a bit more general may be useful: for strings, the duplication of inner loops which essentially do exactly the same thing just with a different function was my main complaint (see two-but-last to-do item in gh-25693). It would really help to make this something declarative (as it was for the the legacy ufuncs). Since I think this is one of the first in-depth uses of the new array method mechanism, maybe it is still a good time to address?

I am very happy to add a slot to the ArrayMethod creation that allows adding static data. And probably exposing it through self. Internally, you could just do that. Making it public (i.e. exposing some of the ArrayMethod struct), is also totally fine for obvious stuff (like this one).

Attaching things to the ufunc is plausible, but right now the ufunc has a map of dtypes -> arraymethod, and making that dtypes -> (arraymethod, void *) is a bit meeh in my eyes.

In other words, I am very happy if we add it to the ArrayMethod and also if we make it public. We could pass it as part of context rather than context->meth->static_data, but attaching it the ufunc (which at least "feels" leaner) seems tedious.

Yes, agreed, attached to the ArrayMethod makes most sense.

Looking at https://docs.python.org/3/c-api/type.html#c.PyType_Slot, typically a slot contains a function, so do you mean adding a slot for a function that provides the data? Or a slot that provides the data directly? (I think you imply the function, but just in case). And would you then use that slot (if defined), in npy_default_get_strided_loop (which is where currently, out_transferdata is set to NULL)?

Or a slot that provides the data directly?

In almost all cases, I like functions, but I was meaning a void *. A function might make sense to me if and only if:

• You use it to fill a new slot in context, and
• the function gets some input.

And I think at least the second just won't be the case (or isn't worthwhile).

OK, so closer to the existing auxdata then!

@seberg - I used this PR for a proof of concept of adding aux data - does it make any sense? Obviously, if it does, we should take it out of this PR...

I went ahead and did all functions that could be fairly easily factored together... About 1000 fewer lines of code.

p.s. The overall diff of stringdtype_ufuncs.cpp is rather dreadful. May be best to view individual commits.

Awesome, thanks for taking this on! I will look this over next week.

@lysnikolaou you probably want to look at this too

By the way, I'm making the DType API public in another PR, so maybe wait on that one being merged for a final round on this PR because there will definitely be conflicts.

OK, this seems to work and I'm fairly happy with it. A general question is whether the dtype C API addition should be split off, and, if so, a specific one whether it might still be made part of gh-25754 (if not, then some guidance on what other files to adjust would be welcome...).

For the ufuncs themselves, the diff of the whole PR is basically unreviewable, but it's OK if done commit by commit. Happy to split it up in different PRs is that feels better.

OK great, I'm hoping to have the dtype API PR done today so hopefully this won't take more than a couple days to sort out. Hope that's OK with you too @lysnikolaou!

No such thing as a major release of a nearly 20 year old open source project without gnrly merge conflicts when we get down to the line!

Yes, n many ways it is a compliment to all those before us that there are so few major merge conflicts!

Right: For the record, I am happy with the addition, I am not 100% on how much of the PyArrayMethod struct we should make public, but up to there seems not unreasonable.

I was mostly hoping to delay these things until Nathans PR is in (and making the descriptor int64 and hiding the fields is gnarly business, even if hopefully not all that bad for downstream).

OK great, I'm hoping to have the dtype API PR done today so hopefully this won't take more than a couple days to sort out. Hope that's OK with you too @lysnikolaou!

Sure! I'm ready to try this for fixed-length dtypes as well as soon as it's merged.

I don't really want to deal with making the PyArrayMethodObject partially public, so for now I'm making NPY_METH_static_data private and prepending the name with an underscore so this is mergeable without resolving the public API.

Ah, oops, we clearly were pushing at the same time, sorry! I like your suggestion - since I messed this up, let me just push from my branch...

Sorry for force-pushing to this branch earlier, I thought you wanted me to resolve the merge conflicts

You had offered that very kindly, but I thought since I had seen the docs going in, I could just do it myself - sorry to work at cross purposes!

OK, should be all OK now. Reminder that for review of the ufunc changes, it makes most sense to go commit by commit - the difference of the whole set is basically unintelligible.

Great! I hope that with this example, refactoring the other PRs will be straightforward.

If you are all happy, we should just put this in, thanks.