Looking at this PR 👀

Nice results on the ruby-xor microbenchmark 👍

Do you think it would make sense to try to apply this optimization to more cfuncs before merging?

I can apply this to String#getbyte and String#byteslice to get rid of all existing "give up because it's not leaf" scenarios. Other existing specialized cfuncs don't need this. This is meant to unblock the specialization of more C funcs that are leaf unless arguments are malformed, and I want to work on new specializations in separate PRs.

Also, can this be applied to all calls to cfuncs in gen_send_cfunc eventually? That would make a big performance difference if possible (doesn't need to be done in this PR).

Theoretically, it's possible. However, given that frame outlining ended up slowing down the interpreter, I'd like to minimize the number of hooks where we push a lazy frame. Right now, num_lazy_frame_check is about 20 on both railsbench and lobsters (0 on chunky-png), meaning it's not called during benchmark iterations.

So the benefit of the current design is that it has zero overhead to the C code, which likely doesn't hold when you apply this to every single C func. We currently apply the hook to rb_to_int with non-integers, which is fairly uncommon, and rb_raise and FrozenError, which can often be a bug that's not shipped to production. It doesn't touch rb_funcall.

But I think this is enough. On chunky-png, num_send_cfunc_inline % was pushed from 76.6% to 99.4% by inlining #setbyte (no significant impact on the speedup though). So it inlines almost everything. By dealing with such C funcs, I believe we can cover a large portion of C calls on most benchmarks. Not-usually-leaf C funcs should be rewritten in Ruby.

I can apply this to String#getbyte and String#byteslice to get rid of all existing "give up because it's not leaf" scenarios

Done 340cfb5 c4bd35e. For String#byteslice, I added -1 argc support too.