BUG, Benchmark: fix passing optimization build options to asv #17736

seiko2plus · 2020-11-09T02:04:28Z

Fix passing optimization build options to asv

closes #17716

see #17716 (comment)

Qiyu8 · 2020-11-09T03:53:39Z

I tested your code in two strategies:

single branch benchmark: create new branch, download your code, run the following command.

python runtests.py --cpu-baseline="sse2" --bench bench_linalg
- The NPY_SIMD is 128.
python runtests.py --cpu-baseline="avx2" --bench bench_linalg
- The NPY_SIMD is 256.
two branch benchmark: create two new branch, download your code in both branch, run the following command.

python runtests.py --cpu-baseline="sse2" --bench-compare test_asv bench_linalg
- The NPY_SIMD is 128.
python runtests.py --cpu-baseline="avx2" --bench-compare test_asv bench_linalg
- The NPY_SIMD is 256.

To sum it up, the code here works fine in my computer. Thanks @seiko2plus .

seiko2plus · 2020-11-09T04:03:40Z

@Qiyu8, thank you for reporting, you can also count on #17737 to verify the supported features. it provides the same format that used within pytest.

mattip · 2020-11-09T06:02:42Z

benchmarks/asv_compare.conf.json.tpl

-        "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"
+        // pip ignores '--global-option' when pep517 is enabled, we also enabling pip verbose to
+        // be reached from asv `--verbose` so we can verify the build options.
+        "PIP_NO_BUILD_ISOLATION=false python {build_dir}/benchmarks/asv_pip_nopep517.py -v {numpy_global_options} --no-deps --no-index -w {build_cache_dir} {build_dir}"


This should be broken into two steps: one to check that --no-use-pep517 is supported, and one to build the wheel. It might be easier to avoid the pip build altogether by adding a bdist_wheel build step

rm -rf dist python setup.py build {numpy_build_options} bdist_wheel python -mpip install dist/*.whl

This should be broken into two steps: one to check that --no-use-pep517 is supported, and one to build the wheel

already handle it through a separate python script asv_pip_nopep517.py, we're not free to use POSIX commands since we're targeting non-unix platforms too. e.g. windows

It might be easier to avoid the pip build altogether by adding a bdist_wheel build step

I tried to use bdist_wheel but ASV wouldn't able to fetch the wheel file and cache it.
also, I tried to only use the pip command only but idk why ccache doesn't work properly

here is the default values of build commands :

"install_command": ["python -mpip install {wheel_file}"], "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"], "build_command": ["python setup.py build", "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"]

Qiyu8 · 2020-11-12T01:54:57Z

I think this PR is good to merge and will unlock other performance optimizations.

mattip · 2020-11-12T06:22:17Z

While I do not debate the value in passing optimization flags to asv, I think this PR still is not ready. There should be a simpler and more direct way to get interaction between asv and code building.

Here are some ideas, it may be worth discussing them more widely:

overhaul runtests.py and refactor it into proper subcommands rather than using home-grown argparse enhancements. This would allow refactoring the code, which might clarify the interactions between building NumPy and running benchmarks, and reveal a better path for this kind of code.
enhance ASV and/or the interaction between the benchmarks, ASV, and NumPy. PRs MAINT, Benchmark: print the supported CPU features during the run of performance tests #17737, ENH: Standalone benchmark script for the inner loops of ufunc #15987 and ENH: Provides a deep benchmark for universal functions #15992 are also related to this.

seiko2plus · 2020-11-12T17:19:40Z

@mattip,

There should be a simpler and more direct way to get interaction between asv and code building.

unfortunately asv continuous doesn't offer any other way, check airspeed-velocity/asv#803

Here are some ideas, it may be worth discussing them more widely

First, we need to take a decision to either accepting this pull request as a workaround or close it and revert #17284

overhaul runtests.py and refactor it into proper subcommands rather than using home-grown argparse enhancements.

as a start, we can create an independent script benchmark.py that mapping all important args to ASV, it should
provides similar functionality to gh-15987

enhance ASV and/or the interaction between the benchmarks, ASV, and NumPy. PRs gh-17737, gh-15987 and gh-15992 are also related to this.

Here is the issue, ASV is based on command arguments that kill the flexibility.
I vote to create our own micro-benchmarking tool and drop ASV.

EDIT: I mentioned a wrong pr

Qiyu8 · 2020-11-16T03:40:59Z

I think It's hard time to drop asv support, because this benchmark toolkit is working well among normal modules, The current arg based simd mechanism exposed the drawbacks of asv, which should improved through community.

seiko2plus · 2020-11-16T04:04:53Z

@Qiyu8, with all respect, we can't count on another project or community in every single new feature. I see google benchmark as a good alternative, we can build our new tool on top of it. It already provides decent flexibility and been used by many projects.

seiko2plus · 2020-12-09T16:08:03Z

@mattip, there's no other way to get ride of it without this workaround. ugly but necessary and the same thing for #17737.
please I need to hear your final call.

mattip · 2020-12-09T17:03:59Z

I marked this for triage-review so we can get some more opinions.

seiko2plus · 2020-12-09T17:07:40Z

@mattip, thank you Matti, I really appreciate your help.

mattip · 2020-12-09T17:13:40Z

The SIMD optimizations is stretching the current workflows:

a need to benchmark ufunc inner loops
a new templating engine for C code
a new compiler option extension for setting machine options to C flags (merged)

Maybe we should think a bit about separating the loops out into a separate library that would be easier to test and benchmark, along the lines of issue gh-17698.

mattip · 2020-12-16T19:55:45Z

In the triage meeting we decided to merge this, but would like to rethink benchmarking in general. That is a different discussion though, so in this goes.

mattip · 2020-12-16T19:56:06Z

Thanks @seiko2plus

seiko2plus · 2020-12-18T03:37:15Z

@mattip,

Infrastructure, SIMD, and GPU optimizations must be part of the core of any modern computing library, branching the core or counting on other projects will badly affect workflow, especially for big projects.

We have something new here, let's give it the full chance before thinking of separating it into a separate repo. I think NumPy should find a foot in highly computing processing instead of playing the mediator role, in order to maintain its popularity.

need to benchmark ufunc inner loops

We need also a tool for sanity data check too. OpenCV has 2 in 1 tool benchmarking, and sanity checks at the same time. see HowToUsePerfTests very effective in detecting any deviations.

a new templating engine for C code

Why do we need a new engine and we have python? we just need a better way to using it in our C sources. I think #17952 can be a good solution.

In the triage meeting we decided to merge this,

Thank you a lot.

but would like to rethink benchmarking in general.

I'm not sure what we should do. To create our own minimal tool that serving our only needs or contributing to ASV. What do you prefer?

mattip · 2020-12-18T08:01:24Z

In general, I do not like re-inventing the wheel. If a tool or paradigm can be adapted to our needs, then that is preferable over creating our own, unless our needs are very different.

I think NumPy should find a foot in highly computing processing instead of playing the mediator role, in order to maintain its popularity.

I disagree. If by splitting out the great work you have done with SIMD we enable others to reuse it, that would be progress.

What do you prefer?

It seems like right now ASV and pyperf are very similar, but neither serves our needs very well. I think we should explore ways to make ASV better before creating our own tool. Making those tools better would help others who are also struggling with benchmarking.

rossbar · 2020-12-18T18:19:51Z

It seems like right now ASV and pyperf are very similar, but neither serves our needs very well. I think we should explore ways to make ASV better before creating our own tool. Making those tools better would help others who are also struggling with benchmarking.

Well said, I very much agree 👍

seiko2plus · 2020-12-18T20:46:32Z

@mattip, I will give ASV a try I guess we have enough burden and no need to open a new front.

BUG, Benchmark: fix passing optimization build options to asv

11728e8

github-actions bot added the 00 - Bug label Nov 9, 2020

seiko2plus mentioned this pull request Nov 9, 2020

SIMD: The --cpu-baseline is not correctly configured in benchmark system. #17716

Closed

seiko2plus mentioned this pull request Nov 9, 2020

MAINT, Benchmark: print the supported CPU features during the run of performance tests #17737

Merged

mattip reviewed Nov 9, 2020

View reviewed changes

mattip added 01 - Enhancement triage review Issue/PR to be discussed at the next triage meeting labels Dec 9, 2020

seiko2plus mentioned this pull request Dec 14, 2020

SIMD: Optimize the performance of np.packbits in AVX2/AVX512F/VSX. #17102

Merged

mattip merged commit d7a75e8 into numpy:master Dec 16, 2020

seiko2plus deleted the issue_17716 branch January 9, 2021 16:52

rgommers added the component: SIMD Issues in SIMD (fast instruction sets) code or machinery label Jul 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG, Benchmark: fix passing optimization build options to asv #17736

BUG, Benchmark: fix passing optimization build options to asv #17736

seiko2plus commented Nov 9, 2020 •

edited

Loading

Qiyu8 commented Nov 9, 2020

seiko2plus commented Nov 9, 2020

mattip Nov 9, 2020

seiko2plus Nov 10, 2020

Qiyu8 commented Nov 12, 2020

mattip commented Nov 12, 2020

seiko2plus commented Nov 12, 2020 •

edited

Loading

Qiyu8 commented Nov 16, 2020

seiko2plus commented Nov 16, 2020

seiko2plus commented Dec 9, 2020

mattip commented Dec 9, 2020

seiko2plus commented Dec 9, 2020

mattip commented Dec 9, 2020

mattip commented Dec 16, 2020

mattip commented Dec 16, 2020

seiko2plus commented Dec 18, 2020

mattip commented Dec 18, 2020

rossbar commented Dec 18, 2020

seiko2plus commented Dec 18, 2020

BUG, Benchmark: fix passing optimization build options to asv #17736

BUG, Benchmark: fix passing optimization build options to asv #17736

Conversation

seiko2plus commented Nov 9, 2020 • edited Loading

Fix passing optimization build options to asv

Qiyu8 commented Nov 9, 2020

seiko2plus commented Nov 9, 2020

mattip Nov 9, 2020

Choose a reason for hiding this comment

seiko2plus Nov 10, 2020

Choose a reason for hiding this comment

Qiyu8 commented Nov 12, 2020

mattip commented Nov 12, 2020

seiko2plus commented Nov 12, 2020 • edited Loading

Qiyu8 commented Nov 16, 2020

seiko2plus commented Nov 16, 2020

seiko2plus commented Dec 9, 2020

mattip commented Dec 9, 2020

seiko2plus commented Dec 9, 2020

mattip commented Dec 9, 2020

mattip commented Dec 16, 2020

mattip commented Dec 16, 2020

seiko2plus commented Dec 18, 2020

mattip commented Dec 18, 2020

rossbar commented Dec 18, 2020

seiko2plus commented Dec 18, 2020

seiko2plus commented Nov 9, 2020 •

edited

Loading

seiko2plus commented Nov 12, 2020 •

edited

Loading