Skip to content

MAINT: Translate binsearch.c.src to C++ using templates. #20489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 14, 2021

Conversation

serge-sans-paille
Copy link
Contributor

Side effects:

  • make each type tag aware of how to compare them
  • add a few __cplusplus guard in numpy header
  • handle confict between some static and non static symbols with the same name

@serge-sans-paille
Copy link
Contributor Author

The Pypy error seems unrelated :-/

@mattip
Copy link
Member

mattip commented Nov 30, 2021

The Pypy error seems unrelated :-/

xref #20491, probably picking up an unstable nightly build

@scratchmex
Copy link
Contributor

scratchmex commented Dec 2, 2021

Could template of C++ replace most of the .c.src code? What are the drawbacks?

@serge-sans-paille
Copy link
Contributor Author

@scratchmex that's indeed the goal, see #19713

const char *sort, char *ret,
npy_intp arr_len, npy_intp key_len,
npy_intp arr_str, npy_intp key_str,
npy_intp sort_str, npy_intp ret_str,
PyArrayObject *NOT_USED)
Cmp cmp)
Copy link
Member

@seiko2plus seiko2plus Dec 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Cmp cmp)

static pointers require at least optimization level 2 in order to be inlined, it's better to use lambda or just pass meta-type as template arg holds operator () contains the scalar operation. for example:

template<typename Tag>
struct cmp_lt {
    template <typename T>
    bool operator () (T a, T b)
    { return a < b; }
};

template<>
struct cmp_lt<half_tag> {
    bool operator () (npy_half a, npy_half b)
    { return HALF_LT(a, b); }
};
// specialize the rest float/cfloat
// ...

to pass it

argbinsearch_<half_tag, cmp_lt>(...);

to initialize it

template<class Tag, template<typename> class Cmp, class T = typename Tag::type>
static int argbinsearch_(...)
{
    Cmp<Tag> cmp;
}

edit: fix template arg

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually very compiler specific, see https://godbolt.org/z/7TGKvM8o3
I've yet to see a case where -O1 is used in a place where optimization really matters, so I'm not a big fan of this argument. at -O2 all compiler inline the function pointer correctly, and I tend to prefer readability over optimizing for -O1. What are the thoughts of the other reviewers on that point?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see -O1 being used for debugging. -O3 breaks occsionally, but -O2 usually fixes it. That said, having unstated assumptions floating around could be dangerous. Is it the case that inlining is not required for the code to run correctly? If so, might add a comment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't change the smeantic of the code, it only changes inlining, i.e. execution speed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually very compiler specific, see https://godbolt.org/z/7TGKvM8o3

on GCC -O1 https://godbolt.org/z/4b6f3hxcM isn't going to inline static pointers.

I tend to prefer readability over optimizing

passing a function pointer for internal use that can be handled in compile time is 100% against readability, 100% against the idea behind the existence of C++. The above suggestion is a common pattern we tend to use it since C++ 98 and it can be replaced with lambda.

@charris,

I can see -O1 being used for debugging

we use -O0 for debugging, distros may build NumPy with -O1.

It doesn't change the smeantic of the code, it only changes inlining, i.e. execution speed.

giving a chance for the compiler to avoid inlining a function with one line expression that can be executed within a loop is in general a sign of bad design.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought BOOL_LT was used else where, which is why I reused existing symbol: to prevent duplication. it turns out it's not the case, so I'll move the definition to the tags.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's actually used in heap sort. To avoid duplication, I'll keep the current implementation until it's not needed elsewhere. Small steps!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding extern C to npysort_common.h will fix the duplication between C and C++ symbols without the need of wraps around via static pointers.

@serge-sans-paille serge-sans-paille force-pushed the feature/road-to-cxx-binsearch branch 2 times, most recently from eaffe98 to 6bc519c Compare December 4, 2021 09:23
@serge-sans-paille
Copy link
Contributor Author

Folks, any thoughts on this new version?

@serge-sans-paille serge-sans-paille force-pushed the feature/road-to-cxx-binsearch branch from 6bc519c to d1c9c58 Compare December 8, 2021 20:32
@serge-sans-paille
Copy link
Contributor Author

(rebased on main)

@serge-sans-paille serge-sans-paille force-pushed the feature/road-to-cxx-binsearch branch from d1c9c58 to 2db8bfb Compare December 8, 2021 20:57
@seiko2plus seiko2plus added the C++ Related to introduction and use of C++ in the NumPy code base label Dec 10, 2021
Comment on lines 257 to 270
template<arg_t arg>
struct binsearch_base;

template<>
struct binsearch_base<arg> {
using function_type = PyArray_ArgBinSearchFunc*;
struct value_type {
int typenum;
function_type binsearch[NPY_NSEARCHSIDES];
};
template<class... Tags>
static constexpr std::array<value_type, sizeof...(Tags)> make_binsearch_map(npy::taglist<Tags...>) {
return std::array<value_type, sizeof...(Tags)>{value_type{Tags::type_value, { (function_type)&argbinsearch<Tags, left>,(function_type)argbinsearch<Tags, right>}}...};
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice !, would you please sort out the code style here?

@seiko2plus
Copy link
Member

@serge-sans-paille,

Folks, any thoughts on this new version?

It looks way better than the previous one, easier to read, no more C macros. Good job!.

Side effects:
- make each type tag aware of how to compare them
- add a few __cplusplus guard in numpy header
- handle confict between some static and non static symbols with the same name
@serge-sans-paille serge-sans-paille force-pushed the feature/road-to-cxx-binsearch branch from 2db8bfb to b358604 Compare December 12, 2021 07:40
@serge-sans-paille
Copy link
Contributor Author

@seiko2plus updated with formatting and usage of wrapper instead of function pointers

@mattip
Copy link
Member

mattip commented Dec 13, 2021

I wonder why coverage thinks some of the code paths are not covered. Is that a false-positive?

@serge-sans-paille
Copy link
Contributor Author

@mattip I think it indicates that binsearch is not tested for all scalar types - the comparison functions are used elsewhere in the C code base so their coverage is 100%, but the C++ version are (currently) only used in binsearch.

Two paths here: either we test all dtypes for binsearch, or we wait until I convert other sort algorithms. Your call ;-)

@mattip
Copy link
Member

mattip commented Dec 13, 2021

I think we can wait for the rest of sort. @seiko2plus any reason not to merge?

@charris
Copy link
Member

charris commented Dec 13, 2021

Do we want to run clang-format on the files? We have the needed .clang-format.

@charris
Copy link
Member

charris commented Dec 13, 2021

For test coverage, there are likely many numpy tests that could be improved by testing over more dtypes.

@seiko2plus
Copy link
Member

@mattip, I have some minor reservations, not sure if it deserves another round of review. In general, the new code looks good to me and we should merge it.

@mattip mattip merged commit 5cd043c into numpy:main Dec 14, 2021
@mattip
Copy link
Member

mattip commented Dec 14, 2021

Thanks @serge-sans-paille

@charris charris changed the title Convert binsearch.c.src to binsearch.cpp MAINT: Translate binsearch.c.src to C++ using templates. Jan 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
03 - Maintenance C++ Related to introduction and use of C++ in the NumPy code base component: numpy._core
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants