-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Add sort.py to microbenchmarks #6086
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
WalkthroughBenchmarks README updated to document subset benchmarking via name matching (e.g., Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the "Integrations" page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
🧹 Nitpick comments (1)
benches/README.md (1)
10-16
: Extend example to include list.sort() microbench and show exact-match usageOnce you add the in-place list.sort() microbenchmark, surface it here. Also consider an exact-match example to avoid unintentionally matching other benchmarks.
Apply:
```shell -cargo bench sort +cargo bench sort +# Or, for the in-place list.sort() microbenchmark: +cargo bench list_sort +# For an exact match on just "sort": +cargo bench '^sort$'</blockquote></details> </blockquote></details> <details> <summary>📜 Review details</summary> **Configuration used: .coderabbit.yml** **Review profile: CHILL** **Plan: Pro** <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between a9a9e3bf118f6c631ef99fe7174215254f846097 and d96712bd950005d8d813bd3a4d8f44fb0426ec58. </details> <details> <summary>📒 Files selected for processing (2)</summary> * `benches/README.md` (1 hunks) * `benches/microbenchmarks/sort.py` (1 hunks) </details> <details> <summary>🧰 Additional context used</summary> <details> <summary>📓 Path-based instructions (1)</summary> <details> <summary>**/*.py</summary> **📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)** > `**/*.py`: Follow PEP 8 style for custom Python code > Use ruff for linting Python code Files: - `benches/microbenchmarks/sort.py` </details> </details><details> <summary>🧠 Learnings (1)</summary> <details> <summary>📚 Learning: 2025-07-19T03:16:56.511Z</summary>
Learnt from: CR
PR: RustPython/RustPython#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-07-19T03:16:56.511Z
Learning: Generate documentation withcargo doc --no-deps --all
**Applied to files:** - `benches/README.md` </details> </details><details> <summary>🧬 Code Graph Analysis (1)</summary> <details> <summary>benches/microbenchmarks/sort.py (2)</summary><blockquote> <details> <summary>stdlib/src/random.rs (1)</summary> * `random` (42-45) </details> <details> <summary>vm/src/stdlib/builtins.rs (1)</summary> * `sorted` (815-820) </details> </blockquote></details> </details><details> <summary>🪛 Flake8 (7.2.0)</summary> <details> <summary>benches/microbenchmarks/sort.py</summary> [error] 3-3: undefined name 'ITERATIONS' (F821) </details> </details> </details> <details> <summary>⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)</summary> * GitHub Check: Run snippets and cpython tests (macos-latest) * GitHub Check: Run snippets and cpython tests on wasm-wasi * GitHub Check: Run snippets and cpython tests (windows-latest) * GitHub Check: Run snippets and cpython tests (ubuntu-latest) * GitHub Check: Run tests under miri * GitHub Check: Check the WASM package and demo * GitHub Check: Check Rust code with rustfmt and clippy * GitHub Check: Run rust tests (ubuntu-latest) * GitHub Check: Run rust tests (windows-latest) * GitHub Check: Run rust tests (macos-latest) * GitHub Check: Ensure compilation on various targets </details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
d96712b
to
5531ae2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
🧹 Nitpick comments (1)
benches/microbenchmarks/sort.py (1)
1-1
: Ensure Consistent RNG Seeding Across MicrobenchmarksNo other benchmarks in
benches/microbenchmarks/
currently seed the RNG. If you’d like deterministic inputs for easier cross-run comparisons, consider adding a fixed seed at the top ofbenches/microbenchmarks/sort.py
:-from random import random +from random import random, seed +seed(0)This change will align
sort.py
with a reproducible input setup—apply only if it matches the rest of your benchmarking strategy.
📜 Review details
Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
benches/README.md
(1 hunks)benches/microbenchmarks/sort.py
(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- benches/README.md
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)
**/*.py
: Follow PEP 8 style for custom Python code
Use ruff for linting Python code
Files:
benches/microbenchmarks/sort.py
🧠 Learnings (3)
📓 Common learnings
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:7-7
Timestamp: 2025-08-09T22:55:10.479Z
Learning: In RustPython benchmarks, when testing sorting performance, benchmarking only `sorted()` is sufficient since it internally uses the same sorting logic as `list.sort()`. The only difference is that `sorted()` creates a copy first, so testing both would be redundant for evaluating the sorting algorithm's performance.
📚 Learning: 2025-08-09T22:55:10.479Z
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:7-7
Timestamp: 2025-08-09T22:55:10.479Z
Learning: In RustPython benchmarks, when testing sorting performance, benchmarking only `sorted()` is sufficient since it internally uses the same sorting logic as `list.sort()`. The only difference is that `sorted()` creates a copy first, so testing both would be redundant for evaluating the sorting algorithm's performance.
Applied to files:
benches/microbenchmarks/sort.py
📚 Learning: 2025-08-09T22:56:24.504Z
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:1-3
Timestamp: 2025-08-09T22:56:24.504Z
Learning: In RustPython's microbenchmarks (benches/microbenchmarks/*.py), the variable `ITERATIONS` is intentionally used without being defined in the Python files. It is injected by the cargo bench harness at runtime. This pattern should be maintained for consistency across all microbenchmarks, and F821 lint warnings for undefined `ITERATIONS` are expected and acceptable in this context.
Applied to files:
benches/microbenchmarks/sort.py
🧬 Code Graph Analysis (1)
benches/microbenchmarks/sort.py (2)
stdlib/src/random.rs (1)
random
(42-45)vm/src/builtins/list.rs (1)
sort
(323-337)
🪛 Flake8 (7.2.0)
benches/microbenchmarks/sort.py
[error] 3-3: undefined name 'ITERATIONS'
(F821)
🔇 Additional comments (2)
benches/microbenchmarks/sort.py (2)
3-3
: ITERATIONS is intentionally harness-injected; keep as-isAcknowledging the established pattern in RustPython microbenchmarks: ITERATIONS is provided by the bench harness. F821 from local linters is expected here and acceptable. No change requested.
7-7
: No change required: setup is re-executed per iteration
The harness uses Criterion’siter_batched_ref
/iter_batched
APIs with thebench_setup
closure, which runs the setup block (includingunsorted_list = […]
) before each sample. That guarantees each sort operates on a fresh, random list.• In
benches/microbenchmarks.rs
, see lines 78–86 (b.iter_batched_ref(|| bench_setup(Some(*idx)), …)
) and lines 147–155 (b.iter_batched(|| bench_setup(Some(*idx)), …)
)
• Each invocation of the setup closure re-executes the code before timing, so no iteration ever sorts an already-sorted list.
Setting the seed is pretty reasonable actually. Will do this next time I'm at my computer |
5531ae2
to
ccc340d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (2)
benches/microbenchmarks/sort.py (2)
4-4
: ITERATIONS is intentionally harness-injected; F821 can be ignored herePer the microbench pattern in this repo, ITERATIONS is provided by the harness. Keeping it undefined in the file is consistent and expected.
8-8
: In-place sort mutates the input; subsequent timed invocations will sort an already-sorted listBecause setup runs once and only the section after “# ---” is timed repeatedly, mutating
unsorted_list
will cause later iterations to measure best-case sorting of an already-sorted list, skewing results. Prefer a non-mutating call or copy within the timed section.Apply one of the following:
Option A (preferred, aligns with “benchmark only sorted()” learning):
-unsorted_list.sort() +sorted(unsorted_list)Option B (if you explicitly want to time list.sort()):
-unsorted_list.sort() +unsorted_list.copy().sort()
🧹 Nitpick comments (1)
benches/README.md (1)
17-17
: Nit: small wording/style improvement and consistency for shell blocks
- Prefer “benchmark with” over “bench against” for clarity.
- Keep shell blocks consistent: either include a
$
prompt in both code blocks or neither (the earlier “cargo bench sort” block has no$
).
📜 Review details
Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
benches/README.md
(1 hunks)benches/microbenchmarks/sort.py
(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)
**/*.py
: Follow PEP 8 style for custom Python code
Use ruff for linting Python code
Files:
benches/microbenchmarks/sort.py
🧠 Learnings (3)
📓 Common learnings
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:7-7
Timestamp: 2025-08-09T22:55:10.512Z
Learning: In RustPython benchmarks, when testing sorting performance, benchmarking only `sorted()` is sufficient since it internally uses the same sorting logic as `list.sort()`. The only difference is that `sorted()` creates a copy first, so testing both would be redundant for evaluating the sorting algorithm's performance.
📚 Learning: 2025-08-09T22:55:10.512Z
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:7-7
Timestamp: 2025-08-09T22:55:10.512Z
Learning: In RustPython benchmarks, when testing sorting performance, benchmarking only `sorted()` is sufficient since it internally uses the same sorting logic as `list.sort()`. The only difference is that `sorted()` creates a copy first, so testing both would be redundant for evaluating the sorting algorithm's performance.
Applied to files:
benches/README.md
benches/microbenchmarks/sort.py
📚 Learning: 2025-08-09T22:56:24.527Z
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:1-3
Timestamp: 2025-08-09T22:56:24.527Z
Learning: In RustPython's microbenchmarks (benches/microbenchmarks/*.py), the variable `ITERATIONS` is intentionally used without being defined in the Python files. It is injected by the cargo bench harness at runtime. This pattern should be maintained for consistency across all microbenchmarks, and F821 lint warnings for undefined `ITERATIONS` are expected and acceptable in this context.
Applied to files:
benches/microbenchmarks/sort.py
🪛 LanguageTool
benches/README.md
[grammar] ~10-~10: There might be a mistake here.
Context: ...ng to run a subset of the benchmarks. To run only the sort microbenchmark, you ca...
(QB_NEW_EN)
[grammar] ~17-~17: There might be a mistake here.
Context: ...rgo bench`. You can also bench against a specific installed Python version by run...
(QB_NEW_EN)
🪛 Flake8 (7.2.0)
benches/microbenchmarks/sort.py
[error] 4-4: undefined name 'ITERATIONS'
(F821)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: Check Rust code with rustfmt and clippy
- GitHub Check: Run snippets and cpython tests (windows-latest)
- GitHub Check: Run snippets and cpython tests (macos-latest)
- GitHub Check: Run snippets and cpython tests (ubuntu-latest)
- GitHub Check: Run rust tests (ubuntu-latest)
- GitHub Check: Ensure compilation on various targets
- GitHub Check: Run rust tests (macos-latest)
- GitHub Check: Run rust tests (windows-latest)
- GitHub Check: Run snippets and cpython tests on wasm-wasi
- GitHub Check: Check the WASM package and demo
🔇 Additional comments (2)
benches/microbenchmarks/sort.py (1)
1-2
: Good call seeding RNG for reproducibilitySeeding ensures deterministic inputs across runs, which is important for comparing interpreters/engines.
benches/README.md (1)
10-16
: Subset benchmark instructions look goodClear, minimal, and consistent with Criterion’s name filtering. Nice addition.
I chose 5 * Iterations to try better show that RustPython sort implementation scales noticeably worse CPython's with respect to the number of elements.
3.13 better reflects the current state of the project vs 3.7.
ccc340d
to
fb3dd8f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (1)
benches/microbenchmarks/sort.py (1)
4-4
: F821 (undefined name ‘ITERATIONS’) is intentional per harness injectionPer prior discussion and existing microbench patterns, ITERATIONS is injected by the harness; no inline default or noqa is required here.
🧹 Nitpick comments (3)
benches/README.md (2)
13-15
: Fix markdownlint MD014: remove leading$
in shell blocksUse plain commands in fenced blocks to satisfy MD014 and improve copy/paste UX.
```shell -$ cargo bench sort +cargo bench sort--- `17-22`: **Minor grammar + markdownlint MD014 cleanups** - Prefer “benchmark against” over “bench against”. - Remove leading “$” in shell example for MD014. ```diff -To view Python tracebacks during benchmarks, run `RUST_BACKTRACE=1 cargo bench`. You can also bench against a +To view Python tracebacks during benchmarks, run `RUST_BACKTRACE=1 cargo bench`. You can also benchmark against a
```shell -$ PYTHON_SYS_EXECUTABLE=python3.13 cargo bench +PYTHON_SYS_EXECUTABLE=python3.13 cargo bench</blockquote></details> <details> <summary>benches/microbenchmarks/sort.py (1)</summary><blockquote> `1-4`: **Avoid mutating global RNG state: use a local Random(0) instance** Seeding the global RNG can affect other benchmarks if they share a process. A local RNG preserves reproducibility without global side effects. ```diff -from random import random, seed -seed(0) +from random import Random +rng = Random(0) -unsorted_list = [random() for _ in range(5 * ITERATIONS)] +unsorted_list = [rng.random() for _ in range(5 * ITERATIONS)]
📜 Review details
Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
benches/README.md
(1 hunks)benches/microbenchmarks/sort.py
(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)
**/*.py
: Follow PEP 8 style for custom Python code
Use ruff for linting Python code
Files:
benches/microbenchmarks/sort.py
🧠 Learnings (3)
📓 Common learnings
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:7-7
Timestamp: 2025-08-09T22:55:10.512Z
Learning: In RustPython benchmarks, when testing sorting performance, benchmarking only `sorted()` is sufficient since it internally uses the same sorting logic as `list.sort()`. The only difference is that `sorted()` creates a copy first, so testing both would be redundant for evaluating the sorting algorithm's performance.
📚 Learning: 2025-08-09T22:55:10.512Z
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:7-7
Timestamp: 2025-08-09T22:55:10.512Z
Learning: In RustPython benchmarks, when testing sorting performance, benchmarking only `sorted()` is sufficient since it internally uses the same sorting logic as `list.sort()`. The only difference is that `sorted()` creates a copy first, so testing both would be redundant for evaluating the sorting algorithm's performance.
Applied to files:
benches/README.md
benches/microbenchmarks/sort.py
📚 Learning: 2025-08-09T22:56:24.527Z
Learnt from: jackoconnordev
PR: RustPython/RustPython#6086
File: benches/microbenchmarks/sort.py:1-3
Timestamp: 2025-08-09T22:56:24.527Z
Learning: In RustPython's microbenchmarks (benches/microbenchmarks/*.py), the variable `ITERATIONS` is intentionally used without being defined in the Python files. It is injected by the cargo bench harness at runtime. This pattern should be maintained for consistency across all microbenchmarks, and F821 lint warnings for undefined `ITERATIONS` are expected and acceptable in this context.
Applied to files:
benches/microbenchmarks/sort.py
🪛 LanguageTool
benches/README.md
[grammar] ~10-~10: There might be a mistake here.
Context: ...ng to run a subset of the benchmarks. To run only the sort microbenchmark, you ca...
(QB_NEW_EN)
[grammar] ~17-~17: There might be a mistake here.
Context: ...rgo bench`. You can also bench against a specific installed Python version by run...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
benches/README.md
14-14: Dollar signs used before commands without showing output
(MD014, commands-show-output)
21-21: Dollar signs used before commands without showing output
(MD014, commands-show-output)
🪛 Flake8 (7.2.0)
benches/microbenchmarks/sort.py
[error] 4-4: undefined name 'ITERATIONS'
(F821)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: Run snippets and cpython tests on wasm-wasi
- GitHub Check: Run rust tests (windows-latest)
- GitHub Check: Run rust tests (macos-latest)
- GitHub Check: Run snippets and cpython tests (windows-latest)
- GitHub Check: Run rust tests (ubuntu-latest)
- GitHub Check: Run snippets and cpython tests (macos-latest)
- GitHub Check: Run snippets and cpython tests (ubuntu-latest)
- GitHub Check: Check Rust code with rustfmt and clippy
- GitHub Check: Check the WASM package and demo
- GitHub Check: Run tests under miri
- GitHub Check: Ensure compilation on various targets
🔇 Additional comments (1)
benches/microbenchmarks/sort.py (1)
8-9
: LGTM: benchmarked code avoids in-place mutationUsing
sorted(unsorted_list)
respects the “setup runs once” constraint and keeps input unchanged across timed runs.
Description
sorted
(and by extensionsort
) builtin functionsI chose
5 * ITERATIONS
to better show the divergence between the sort implementations, while trying to avoid individual microbenchmark runs timing out.Sample benchmark results
Violin plot

Line Chart

Manually benchmark
Using larger list sizes really shows the difference. Sorting 1_000_000 random numbers:
Summary by CodeRabbit
New Features
Documentation