Can we reuse our existing benchmark machinery here?

@eric-wieser, I tried to use ASV but the result wasn't stable enough, check this patch and patch2 from #13516, the idea behind this patch is to benchmarking only the inner loop of ufunc in order to reduce the noises as much as possible, also ASV is kinda slow too.

EDIT: I moved the two mentioned patches to separate pull-requests #15992 and #15990

It would be nice if we could at least hook into ASV for things like benchmark result comparisons and storage, rather than building our own version of those too. It might be worth starting a conversation with @pv about the best way to do that.

@seiko2plus you are repeating the function run multiple times here within your run function. May that be enough to stabilize the results a bit in asv?

EDIT: This got lost: "You are doing a few other things here that you are not doing in the asv version."

For example, if you just define the run function in C (and monkeypatch it into Benchmark), and make it do a couple of C-level calls (to offset the ~200ns or so overhead. That might be enough to get a stable result as well?

@seberg, ASV already collect multiple samples for each benchmark, but still not stable enough even on idle CPU.

This script is not providing a replacement for the current ASV implementation, the main reason behind it is to detect any performance changes in the inner loops of ufunc and removing the functionality of umath and multiarry from the equation in order to reduce the noises as much as possible, it also provides more testing cases like multiple strides, sizes and better control for the testing process.

For example, if you just define the run function in C (and monkeypatch it into Benchmark), and make it do a couple of C-level calls (to offset the ~200ns or so overhead. That might be enough to get a stable result as well?

The problem is ASV doesn't provide a way to specify the elapsed time manually.

EDIT: This got lost: "You are doing a few other things here that you are not doing in the asv version."

@seberg, I moved the mentioned patches from #13516, into a separate pull #15992 and #15990. also modified the number of repeats and samples to be equal to the default settings of this script.
but still, the ratio of ASV not stable enough.

One reason that could be causing noise is turbo mode. In case you haven't already done, I would recommend disabling for benchmarking purposes (set /sys/devices/system/cpu/intel_pstate/no_turbo to 1). May be that will help? I haven't had too much variability while benchmarking ufunc's with asv.

@r-devulap, Before I run any benchmarks, I usually do:

• isolate logical cores from scheduling through linux kernel options isolcpus and rcu_nocbs
• reducing scheduling-clock ticks through nohz_full for the isolated cores
• use option --cpu-affinity that comes with this script or what ASV provides for the isolated cores
• use scaling governor performance via /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
• disable turbo boost via /sys/devices/system/cpu/intel_pstate/no_turbo
• make sure that ASLR(address space layout randomization) state is 'full randomization' through
  set 2 to /proc/sys/kernel/randomize_va_space

Lately, I realized a python module called pyperf, provides a tool to tune the system with the above tips and many more via command pyperf system tune

However, it seems I should have an idle hardware in order to get almost stable ratios for ASV not just isolate some logical cores since any involved system calls that interpret the thread during collecting the benchmark samples will eliminate the benefits from isolating the logical cores via isolcpus and rcu_nocbs.

One of the things I don't like in ASV that its uses a separate process for each collected sample,
which makes it too slow.

ping @pv. Is there something here that we all are missing?

I ran this PR on a live environment without a desktop (Ubuntu Server), using the method in the PR description. The noise was around 3% and this PR had a performance impact of ±5%, so not too much of a difference.

Whoops, had the wrong tab open. This comment was meant for #16247, copy pasting there.

close/reopen