Skip to content

Support user-defined benchmark suites. #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 127 commits into from
Dec 8, 2021
Merged

Support user-defined benchmark suites. #109

Changes from 1 commit
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
509d93c
benchmarks -> _benchmarks
ericsnowcurrently Jun 18, 2021
e16141d
Use a new API for pyperformance.benchmarks.
ericsnowcurrently Jun 18, 2021
d2101b2
Deal with benchmark objects instead of names.
ericsnowcurrently Jun 21, 2021
20b2c23
Refactor select_benchmarks().
ericsnowcurrently Jun 21, 2021
20e8492
Add and use the default manifest file.
ericsnowcurrently Jun 21, 2021
4f74cb8
Move run_perf_script() back.
ericsnowcurrently Jun 22, 2021
3917d22
Clean up benchmark/__init__.py.
ericsnowcurrently Jun 22, 2021
32e15ec
Make the utils a package.
ericsnowcurrently Jun 22, 2021
b041e2e
Move each of the default benchmarks into its own directory.
ericsnowcurrently Jun 23, 2021
d79eba5
Make BenchmarkSpec.metafile as "secondary" attribute.
ericsnowcurrently Jun 23, 2021
f30421d
Fix benchmark selection.
ericsnowcurrently Jun 23, 2021
043836b
Fix the default benchmarks selection.
ericsnowcurrently Jun 24, 2021
6ba9603
Fix the run script filename.
ericsnowcurrently Jun 24, 2021
0f15e8e
Run benchmarks from the metadata instead of hard-coded.
ericsnowcurrently Jun 24, 2021
91c27f9
Fix the requirements.
ericsnowcurrently Jun 24, 2021
a4f97ad
Pass "name" through to parse_pyproject_toml().
ericsnowcurrently Jun 25, 2021
6d385bf
Leave a note about classifiers.
ericsnowcurrently Jun 25, 2021
5308b29
Drop an unused file.
ericsnowcurrently Jun 26, 2021
4c8a18f
Load manifest and select benchmarks before running the command.
ericsnowcurrently Jun 28, 2021
0f63a54
Fix Benchmark.__repr__().
ericsnowcurrently Jun 28, 2021
c7a92f4
Fix a default arg in load_metadata().
ericsnowcurrently Jun 28, 2021
c45ece7
Fix the packaging data.
ericsnowcurrently Jun 29, 2021
e940e24
Support per-benchmark venvs in VirtualEnvironment.
ericsnowcurrently Jun 29, 2021
0420e96
Ignore pyproject.toml name only if provided.
ericsnowcurrently Jun 30, 2021
6bce386
Add requirements lock files to the benchmarks.
ericsnowcurrently Jun 30, 2021
29cf228
Make a venv for each benchmark instead of sharing one.
ericsnowcurrently Jun 30, 2021
6cadadc
Support "libsdir" in metadata.
ericsnowcurrently Jul 20, 2021
3fb5187
Fix an error message.
ericsnowcurrently Jul 20, 2021
abf56e9
Use the default resolve() if the default manifest is explicit.
ericsnowcurrently Jul 20, 2021
4bf223a
Merge in the version properly.
ericsnowcurrently Jul 20, 2021
9acbbe4
Preserve PYTHONPATH (with libsdir) when invoking pyperf via benchmark…
ericsnowcurrently Jul 20, 2021
d06aa7b
Add a note about using an upstream lib for parsing pyproject.toml.
ericsnowcurrently Jul 20, 2021
e48d29b
Use the full benchmark version rather than the canonicalized form.
ericsnowcurrently Jul 20, 2021
31f6021
Add iter_clean_lines() to _utils.
ericsnowcurrently Jul 22, 2021
4b13645
Use a run ID when running benchmarks.
ericsnowcurrently Jul 22, 2021
b045e9a
Finish implementing "pre" and "post" script support.
ericsnowcurrently Jul 22, 2021
54b8ba0
Drop "pre" and "post" script support. (It isn't necessary.)
ericsnowcurrently Jul 22, 2021
362bb6e
Use the correct venv name for each benchmark.
ericsnowcurrently Jul 22, 2021
e4fc5d4
Stop supporting "libsdir" in the benchmark metadata.
ericsnowcurrently Jul 22, 2021
9816103
Clean up run_perf_script() and related code.
ericsnowcurrently Jul 22, 2021
b7f90f3
Add a missing import.
ericsnowcurrently Sep 30, 2021
7d42a40
Drop accidental files.
ericsnowcurrently Oct 4, 2021
bb599e9
Install all dependencies when running tests.
ericsnowcurrently Oct 4, 2021
df30711
Temporarily get tests passing on Windows.
ericsnowcurrently Oct 4, 2021
8d563f9
Ignore stdlib_dir mismatch for now.
ericsnowcurrently Oct 4, 2021
d05b07c
Move requirements.txt into the data dir.
ericsnowcurrently Nov 2, 2021
701c910
Move the benchmarks to the data dir.
ericsnowcurrently Nov 2, 2021
afafda6
Move _pythoninfo out of the _utils dir.
ericsnowcurrently Nov 2, 2021
cf4679a
Move _pyproject_toml out of the _utils dir.
ericsnowcurrently Nov 2, 2021
d3bdaad
Make _utils a single module.
ericsnowcurrently Nov 2, 2021
8f0b3e3
Move benchmarks.* up to the top level.
ericsnowcurrently Nov 2, 2021
3ce65a6
Move benchmark.* up to the top level.
ericsnowcurrently Nov 2, 2021
3479e27
Clean up the metadata files.
ericsnowcurrently Nov 2, 2021
35e0c7a
Do not import _manifest in itself.
ericsnowcurrently Nov 2, 2021
0fe268a
Drop version and origin from the manifest.
ericsnowcurrently Nov 2, 2021
698dec9
Add a project-level symlink to the default benchmarks.
ericsnowcurrently Nov 2, 2021
f3c6c4b
Names starting with a digit.
ericsnowcurrently Nov 2, 2021
0987842
Drop the base metadata file.
ericsnowcurrently Nov 2, 2021
2c99d0f
All extra project fields even if there is a metabase.
ericsnowcurrently Nov 2, 2021
121a281
Fix a typo.
ericsnowcurrently Nov 2, 2021
0f9d202
Allow specifying the supported groups in the manifest.
ericsnowcurrently Nov 2, 2021
3d19edf
Add tags to all the benchmarks.
ericsnowcurrently Nov 2, 2021
7270c76
Use the tags to get the groups in the default manifest.
ericsnowcurrently Nov 3, 2021
efeb77e
Show the default group before the others.
ericsnowcurrently Nov 3, 2021
3b3de1d
Drop an outdated comment.
ericsnowcurrently Nov 3, 2021
8ceb990
Allow excluding benchmarks in a group.
ericsnowcurrently Nov 3, 2021
0f10859
Finish _init_metadata().
ericsnowcurrently Nov 3, 2021
9e3124d
metabase -> inherits
ericsnowcurrently Nov 3, 2021
62f7c23
Document the manifest and benchmark formats.
ericsnowcurrently Nov 3, 2021
05bb86f
Fix some typos.
ericsnowcurrently Nov 6, 2021
318720f
Print some diagnostic info on error.
ericsnowcurrently Nov 6, 2021
b51042b
Fall back to metadata for version.
ericsnowcurrently Nov 6, 2021
5eeae8e
Add benchmarks to the default group instead of names.
ericsnowcurrently Nov 6, 2021
0dc395a
Install the requirements, even if the venv already exists.
ericsnowcurrently Nov 6, 2021
6fb7403
Only re-install reqs for benchmark venvs.
ericsnowcurrently Nov 8, 2021
4979d5b
Support an "includes" section in the manifest.
ericsnowcurrently Nov 9, 2021
7f7b571
doc fixes
ericsnowcurrently Nov 9, 2021
0addec2
"all" and "default" are always valid groups.
ericsnowcurrently Nov 9, 2021
57e7070
Do not import pyperformance._manifest unless already installed.
ericsnowcurrently Nov 9, 2021
c55d690
Ensure we run in a venv when needed.
ericsnowcurrently Nov 9, 2021
01697d5
Do not re-install the shared venv.
ericsnowcurrently Nov 9, 2021
94ac28a
Only list the "all" and "default" groups once.
ericsnowcurrently Nov 9, 2021
f4b09bb
Add the manifest to the "compile" config.
ericsnowcurrently Nov 9, 2021
b56b25a
Adjust the stdlib_dir check.
ericsnowcurrently Nov 9, 2021
f8338ae
Be sure to set base_executable.
ericsnowcurrently Nov 9, 2021
dd4597b
Use the --venv opt to the "compile" command.
ericsnowcurrently Nov 15, 2021
e22eeb8
Separate the logic for create vs. recreate.
ericsnowcurrently Nov 15, 2021
6bb3a9e
Do not re-create the venv if already running in it.
ericsnowcurrently Nov 15, 2021
9c664ff
Ensure all requirements are always isntalled.
ericsnowcurrently Nov 15, 2021
91c09d3
Do not buffer stdout during tests.
ericsnowcurrently Nov 15, 2021
7a33956
Fix a check.
ericsnowcurrently Nov 15, 2021
62e2014
Do not buffer stdout during tests.
ericsnowcurrently Nov 15, 2021
d7ea256
Always switch to a venv if running out of the repo.
ericsnowcurrently Nov 15, 2021
8ed6fd5
Distinguish message from runtests.py.
ericsnowcurrently Nov 15, 2021
ce6d09e
Pass values into VirtualEnvironment instead of the options object.
ericsnowcurrently Nov 15, 2021
4048199
Do not buffer stdout during tests.
ericsnowcurrently Nov 15, 2021
df8cccf
Distinguish message from runtests.py.
ericsnowcurrently Nov 15, 2021
8719636
Print out the --venv option.
ericsnowcurrently Nov 15, 2021
3c5d7da
Print out the --venv option.
ericsnowcurrently Nov 15, 2021
14761b2
Print out the --venv option.
ericsnowcurrently Nov 15, 2021
b975c51
Do not add args directly to the "venv" command.
ericsnowcurrently Nov 16, 2021
062745b
Be explicit about "create".
ericsnowcurrently Nov 16, 2021
d918429
Drop debug messages.
ericsnowcurrently Nov 16, 2021
5842d48
Resolve the manifest file in the compile config.
ericsnowcurrently Nov 16, 2021
f1f9db1
Add a "dryrun" mode for testing "compile".
ericsnowcurrently Nov 16, 2021
35166b7
Add BenchmarkManifest.show().
ericsnowcurrently Nov 16, 2021
561d271
Use --manifest and --benchmarks when creating venv for "compile".
ericsnowcurrently Nov 16, 2021
6831508
Add the resolve_file() util.
ericsnowcurrently Nov 16, 2021
bb88341
Resolve the manifest file in includes.
ericsnowcurrently Nov 16, 2021
1e51f8e
Default BenchmarkRevision._dryrun to False.
ericsnowcurrently Nov 16, 2021
ef231fe
Set the default for --benchmarks manually.
ericsnowcurrently Nov 16, 2021
648bd18
Allow the "venv" command to not install benchmark requirements.
ericsnowcurrently Nov 16, 2021
7bb8d94
Use <NONE> as a marker for "no benchmarks".
ericsnowcurrently Nov 16, 2021
15fd559
Require --benchmarks (or default) for some commands.
ericsnowcurrently Nov 16, 2021
f041503
Do not always install the first benchmark venv.
ericsnowcurrently Nov 16, 2021
452b541
Separate creating venv from installing requirements.
ericsnowcurrently Nov 16, 2021
b049a4a
Only install per-benchmark requirements when running them.
ericsnowcurrently Nov 16, 2021
a083cd5
Print the benchmark number.
ericsnowcurrently Nov 16, 2021
7b18753
Factor out Python.resolve_program().
ericsnowcurrently Nov 17, 2021
c8a7789
Do not pass --benchmarks when creating venv for "compile".
ericsnowcurrently Nov 17, 2021
6f6df4d
Skip a benchmark if its requirements could not be installed.
ericsnowcurrently Nov 17, 2021
b379536
Set Python.program to None if the resolved path does not exist.
ericsnowcurrently Nov 17, 2021
686a96d
Factor out resolve_python().
ericsnowcurrently Nov 17, 2021
664a909
Add a blank line.
ericsnowcurrently Nov 17, 2021
9899813
Do not print a traceback for skipped benchmarks.
ericsnowcurrently Nov 17, 2021
da1b6c3
Fix a typo.
ericsnowcurrently Nov 17, 2021
09ffb6b
Merge main.
ericsnowcurrently Dec 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use the correct venv name for each benchmark.
  • Loading branch information
@ericsnowcurrently
ericsnowcurrently committed Oct 4, 2021
commit 362bb6edd208889a477d35f9e29f103d80e5478b
4 changes: 2 additions & 2 deletions pyperformance/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ def run_benchmarks(should_run, python, options):
benchmarks = {}
for bench in to_run:
bench_runid = runid._replace(bench=bench)
venv = _venv.VirtualEnvironment(options, bench, runid.name,
usebase=True)
venv = _venv.VirtualEnvironment(options, bench, bench_runid.name,
usebase=True)
venv.create()
benchmarks[bench] = (venv, bench_runid)

Expand Down