So I attempted to set up a way to benchmark a python function with criterion but it seems that it is currently impossible to use criterion in this way because of how criterion uses closures. At least the way I was attempting to use it.

assuming code is the compiled code for a file like

ctx = object()

def setup():
    ctx.items = []

def bench(ctx):
    ctx.items.append(1)

I was attempting to isolate the bench function and run it for benchmarking

let vars = vm.ctx.new_scope(); // Keep track of local variables
let result = vm.run_code_obj(code, vars.clone())?;

let setup_fn = vars.load_name(&vm, "setup");
let bench_fn = vars.load_name(&vm, "bench");
match setup_fn {
    Some(sf) => {
        return vm.invoke(sf, PyFuncArgs::default());
    }
    _ => println!("No setup function for bench '{}'", name),
}
match bench_fn {
    Some(f) => {
        let mut c = Criterion::default().configure_from_args();
        //let name_str = name.clone();
        c.bench_function(name.clone().as_str(), move |b| {
            b.iter(move || vm.invoke(f, PyFuncArgs::default()))
        });
        Ok(vm.get_none())
    }
    _ => {
        let name_error_type = vm.ctx.exceptions.name_error.clone();
        let msg = format!("bench function is not defined");
        let name_error = vm.new_exception(name_error_type, msg);
        Err(name_error)
    }
}

however no matter what I try I can't get code that works like this to compile. the function object just can not be moved into the inner closure.

I'm not sure a Rust-level benchmark runner is needed?

CPython has the entire suite in Python itself, and AFAIK they managed to make it quite consistent (I'm not sure if criterion.rs actually does some of these things):

https://github.com/python/performance
https://github.com/vstinner/perf

CPython has the entire suite in Python itself, and AFAIK they managed to make it quite consistent (I'm not sure if criterion.rs actually does some of these things):

https://github.com/python/performance
https://github.com/vstinner/perf

hmm, yes I think this is actually the best approach.

we'll have to get the standard library, for the most part, working before we can really use it; but the end result is that we can compare not only performance change to change but with other python versions as well.

@OddCoincidence recently someone did this

#2367

would it help in figuring out if some critical bits of code need improvements?