Skip to content

Conversation

mishushakov
Copy link
Member

@mishushakov mishushakov commented Aug 28, 2025

Changeling

  • Added performance test script
  • Added workflow that uploads performance graph to GItHub Artifacts
  • Updated dependencies

Example report

--- Iteration 1/20 ---
Sandbox creation time: 3703.90 milliseconds
Health check time: 1679.42 milliseconds
First code run time: 2012.40 milliseconds
Second code run time: 83.36 milliseconds

--- Iteration 2/20 ---
Sandbox creation time: 3589.60 milliseconds
Health check time: 800.49 milliseconds
First code run time: 1328.35 milliseconds
Second code run time: 78.94 milliseconds

--- Iteration 3/20 ---
Sandbox creation time: 3950.90 milliseconds
Health check time: 990.09 milliseconds
First code run time: 1861.35 milliseconds
Second code run time: 256.31 milliseconds

--- Iteration 4/20 ---
Sandbox creation time: 4179.79 milliseconds
Health check time: 1063.44 milliseconds
First code run time: 2308.52 milliseconds
Second code run time: 90.12 milliseconds

--- Iteration 5/20 ---
Sandbox creation time: 3493.78 milliseconds
Health check time: 1590.05 milliseconds
First code run time: 2136.84 milliseconds
Second code run time: 83.56 milliseconds

--- Iteration 6/20 ---
Sandbox creation time: 5898.87 milliseconds
Health check time: 1427.22 milliseconds
First code run time: 1508.56 milliseconds
Second code run time: 99.18 milliseconds

--- Iteration 7/20 ---
Sandbox creation time: 4445.07 milliseconds
Health check time: 1078.96 milliseconds
First code run time: 2172.44 milliseconds
Second code run time: 87.28 milliseconds

--- Iteration 8/20 ---
Sandbox creation time: 3508.56 milliseconds
Health check time: 3391.37 milliseconds
First code run time: 2915.50 milliseconds
Second code run time: 139.05 milliseconds

--- Iteration 9/20 ---
Sandbox creation time: 359.29 milliseconds
Health check time: 259.16 milliseconds
First code run time: 319.30 milliseconds
Second code run time: 86.48 milliseconds

--- Iteration 10/20 ---
Sandbox creation time: 541.64 milliseconds
Health check time: 242.10 milliseconds
First code run time: 484.59 milliseconds
Second code run time: 88.08 milliseconds

--- Iteration 11/20 ---
Sandbox creation time: 379.57 milliseconds
Health check time: 2860.15 milliseconds
First code run time: 1554.20 milliseconds
Second code run time: 88.19 milliseconds

--- Iteration 12/20 ---
Sandbox creation time: 500.54 milliseconds
Health check time: 371.44 milliseconds
First code run time: 3798.32 milliseconds
Second code run time: 101.46 milliseconds

--- Iteration 13/20 ---
Sandbox creation time: 4777.97 milliseconds
Health check time: 1363.83 milliseconds
First code run time: 1898.27 milliseconds
Second code run time: 85.63 milliseconds

--- Iteration 14/20 ---
Sandbox creation time: 2610.64 milliseconds
Health check time: 2276.11 milliseconds
First code run time: 1469.17 milliseconds
Second code run time: 122.25 milliseconds

--- Iteration 15/20 ---
Sandbox creation time: 3412.06 milliseconds
Health check time: 1524.70 milliseconds
First code run time: 2559.70 milliseconds
Second code run time: 106.23 milliseconds

--- Iteration 16/20 ---
Sandbox creation time: 441.73 milliseconds
Health check time: 199.83 milliseconds
First code run time: 297.81 milliseconds
Second code run time: 87.16 milliseconds

--- Iteration 17/20 ---
Sandbox creation time: 437.14 milliseconds
Health check time: 248.20 milliseconds
First code run time: 783.20 milliseconds
Second code run time: 90.88 milliseconds

--- Iteration 18/20 ---
Sandbox creation time: 509.18 milliseconds
Health check time: 212.74 milliseconds
First code run time: 296.69 milliseconds
Second code run time: 247.35 milliseconds

--- Iteration 19/20 ---
Sandbox creation time: 627.61 milliseconds
Health check time: 351.45 milliseconds
First code run time: 327.99 milliseconds
Second code run time: 140.21 milliseconds

--- Iteration 20/20 ---
Sandbox creation time: 411.33 milliseconds
Health check time: 224.64 milliseconds
First code run time: 338.85 milliseconds
Second code run time: 81.72 milliseconds

==================================================
PERFORMANCE SUMMARY
==================================================

Sandbox Creation Time Summary:
  Low:    359.29 ms
  High:   5898.87 ms
  Mean:   2388.96 ms
  Median: 3011.35 ms

Health Check Time Summary:
  Low:    199.83 ms
  High:   3391.37 ms
  Mean:   1107.77 ms
  Median: 1026.77 ms

First Code Run Time Summary:
  Low:    296.69 ms
  High:   3798.32 ms
  Mean:   1518.60 ms
  Median: 1531.38 ms

Second Code Run Time Summary:
  Low:    78.94 ms
  High:   256.31 ms
  Mean:   112.17 ms
  Median: 89.16 ms

Generating performance plot...
Performance plot saved as: performance_plot.png

Example plot

performance_plot-2

Copy link

linear bot commented Aug 28, 2025

…ration

- Renamed job from 'publish' to 'performance-tests' for clarity.
- Added step to upload performance plot artifact after tests.
- Updated boxplot function to use 'tick_labels' for better label handling.
@mishushakov mishushakov marked this pull request as ready for review August 28, 2025 17:25
@mishushakov mishushakov enabled auto-merge (squash) August 28, 2025 17:25
@mishushakov mishushakov merged commit 5592fd6 into main Aug 29, 2025
8 checks passed
@mishushakov mishushakov deleted the code-interpreter-performance-benchmarks-e2b-2938 branch August 29, 2025 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants