Skip to content

Enabled Pull Request CI Performance info generation and publishing #911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jan 10, 2023
Merged

Enabled Pull Request CI Performance info generation and publishing #911

merged 2 commits into from
Jan 10, 2023

Conversation

filipecosta90
Copy link
Contributor

This PR enables Automated performance analysis summaries to be generated and published to PRs that are labeled with action:run-benchmark label.

@filipecosta90 filipecosta90 requested a review from rafie January 5, 2023 16:23
@filipecosta90
Copy link
Contributor Author

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 27 stable tests between versions.
  • Detected a total of 3 improvements above the improvement water line.
  • Detected a total of 12 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and perf-reporting.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison perf-reporting (median obs. +- std.dev) % change (higher-better) Note
json_arrappend_geojson 64504 +- 5.7% (7 datapoints) 68951 +- nan% (1 datapoints) 6.9% waterline=5.7%. IMPROVEMENT
json_get_ResultSet.totalResultsAvailable_jsonsl-yahoo2_json 135128 +- 5.8% (7 datapoints) 119752 +- nan% (1 datapoints) -11.4% waterline=5.8%. REGRESSION
json_get_[0]_jsonsl-1 131572 +- 1.4% (7 datapoints) 125777 +- nan% (1 datapoints) -4.4% potential REGRESSION
json_get_[7]_jsonsl-1 129022 +- 4.9% (7 datapoints) 114278 +- nan% (1 datapoints) -11.4% REGRESSION
json_get_[8].zero_jsonsl-1 122690 +- 5.0% (7 datapoints) 119752 +- nan% (1 datapoints) -2.4% waterline=5.0%. -- no change --
json_get_[web-app].servlet[0][servlet-name]_json-parser-0000 99985 +- 5.3% (7 datapoints) 102554 +- nan% (1 datapoints) 2.6% waterline=5.3%. -- no change --
json_get_[web-app].servlet[0]_json-parser-0000 83671 +- 4.7% (7 datapoints) 82295 +- nan% (1 datapoints) -1.6% -- no change --
json_get_[web-app].servlet_json-parser-0000 73520 +- 7.1% (7 datapoints) 77209 +- nan% (1 datapoints) 5.0% waterline=7.1%. potential IMPROVEMENT
json_get_array_of_docs[1]_pass_100_json 109282 +- 6.9% (7 datapoints) 112352 +- nan% (1 datapoints) 2.8% waterline=6.9%. -- no change --
json_get_array_of_docs[1]sclr_pass_100_json 122690 +- 6.9% (7 datapoints) 120473 +- nan% (1 datapoints) -1.8% waterline=6.9%. -- no change --
json_get_array_of_docs_pass_100_json 122690 +- 6.0% (7 datapoints) 120861 +- nan% (1 datapoints) -1.5% waterline=6.0%. -- no change --
json_get_fulldoc_json-parser-0000 78730 +- 4.4% (7 datapoints) 74896 +- nan% (1 datapoints) -4.9% potential REGRESSION
json_get_fulldoc_jsonsl-1 94329 +- 4.8% (7 datapoints) 94331 +- nan% (1 datapoints) 0.0% -- no change --
json_get_fulldoc_jsonsl-yahoo2_json 28768 +- 3.8% (7 datapoints) 27388 +- nan% (1 datapoints) -4.8% potential REGRESSION
json_get_fulldoc_jsonsl-yelp_json 14176 +- 2.7% (7 datapoints) 13783 +- nan% (1 datapoints) -2.8% -- no change --
json_get_fulldoc_pass_100_json 121203 +- 5.0% (7 datapoints) 104703 +- nan% (1 datapoints) -13.6% waterline=5.0%. REGRESSION
json_get_key_empty 137923 +- 7.5% (7 datapoints) 130712 +- nan% (1 datapoints) -5.2% waterline=7.5%. potential REGRESSION
json_get_message.code_jsonsl-yelp_json 133326 +- 4.6% (7 datapoints) 116271 +- nan% (1 datapoints) -12.8% REGRESSION
json_get_sclr_pass_100_json 140837 +- 5.2% (7 datapoints) 116271 +- nan% (1 datapoints) -17.4% waterline=5.2%. REGRESSION
json_get_sub_doc.sclr_pass_100_json 136979 +- 4.8% (7 datapoints) 135128 +- nan% (1 datapoints) -1.4% -- no change --
json_get_sub_doc_pass_100_json 127379 +- 6.0% (7 datapoints) 111101 +- nan% (1 datapoints) -12.8% waterline=6.0%. REGRESSION
json_numincrby_num_1 126573 +- 4.3% (7 datapoints) 116951 +- nan% (1 datapoints) -7.6% REGRESSION
json_nummultby_num_2 117636 +- 8.2% (7 datapoints) 131572 +- nan% (1 datapoints) 11.8% waterline=8.2%. IMPROVEMENT
json_recursive_descent_with_filter_uid_issue674 186 +- 2.1% (7 datapoints) 186 +- nan% (1 datapoints) -0.0% -- no change --
json_set_ResultSet.totalResultsAvailable_1_jsonsl-yahoo2_json 123448 +- 7.3% (7 datapoints) 124217 +- nan% (1 datapoints) 0.6% waterline=7.3%. -- no change --
json_set_[0]foo_jsonsl-1 111101 +- 8.1% (7 datapoints) 118335 +- nan% (1 datapoints) 6.5% waterline=8.1%. potential IMPROVEMENT
json_set_[web-app].servlet[0][servlet-name]_bar_json-parser-0000 84378 +- 5.3% (7 datapoints) 82295 +- nan% (1 datapoints) -2.5% waterline=5.3%. -- no change --
json_set_fulldoc_api_replies_q1_google_autocomplete 41310 +- 3.2% (7 datapoints) 42006 +- nan% (1 datapoints) 1.7% -- no change --
json_set_fulldoc_api_replies_q2_gmaps_areatraffic 71932 +- 4.7% (7 datapoints) 66768 +- nan% (1 datapoints) -7.2% REGRESSION
json_set_fulldoc_api_replies_q3_gmaps_passiveassist 38672 +- 1.6% (7 datapoints) 38193 +- nan% (1 datapoints) -1.2% -- no change --
json_set_fulldoc_api_replies_q4_gmaps_assist 90901 +- 4.8% (7 datapoints) 88685 +- nan% (1 datapoints) -2.4% -- no change --
json_set_fulldoc_api_replies_q5_gmaps_place 1498 +- 0.9% (7 datapoints) 1498 +- nan% (1 datapoints) -0.0% -- no change --
json_set_fulldoc_pass-json-parser-0000 31583 +- 3.6% (7 datapoints) 30709 +- nan% (1 datapoints) -2.8% -- no change --
json_set_fulldoc_pass_100_json 80312 +- 2.8% (7 datapoints) 81623 +- nan% (1 datapoints) 1.6% -- no change --
json_set_fulldoc_yahoo2 8052 +- 2.4% (7 datapoints) 7940 +- nan% (1 datapoints) -1.4% -- no change --
json_set_key_empty 145977 +- 5.3% (7 datapoints) 138881 +- nan% (1 datapoints) -4.9% waterline=5.3%. potential REGRESSION
json_set_message.code_1_jsonsl-yelp_json 122690 +- 4.7% (7 datapoints) 110487 +- nan% (1 datapoints) -9.9% REGRESSION
json_set_num_0 141836 +- 6.1% (7 datapoints) 154288 +- nan% (1 datapoints) 8.8% waterline=6.1%. IMPROVEMENT
json_set_sclr_1_pass_100_json 126573 +- 8.1% (7 datapoints) 115599 +- nan% (1 datapoints) -8.7% waterline=8.1%. REGRESSION
json_set_sclr_pass_100_json 129863 +- 5.6% (7 datapoints) 117639 +- nan% (1 datapoints) -9.4% waterline=5.6%. REGRESSION
json_vs_hashes_hset_key_simple 155032 +- 4.6% (7 datapoints) 161285 +- nan% (1 datapoints) 4.0% potential IMPROVEMENT
json_vs_hashes_json.set_key_simple 124992 +- 3.2% (7 datapoints) 117986 +- nan% (1 datapoints) -5.6% REGRESSION

@rafie rafie self-requested a review January 10, 2023 15:59
rafie
rafie previously approved these changes Jan 10, 2023
View reviewed changes
@filipecosta90 filipecosta90 requested a review from rafie January 10, 2023 16:00
@codecov
Copy link

codecov bot commented Jan 10, 2023

Codecov Report

Base: 81.36% // Head: 81.36% // No change to project coverage 👍

Coverage data is based on head (620aa8d) compared to base (5b6a854).
Patch has no changes to coverable lines.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master     #911   +/-   ##
=======================================
  Coverage   81.36%   81.36%           
=======================================
  Files          15       15           
  Lines        3896     3896           
=======================================
  Hits         3170     3170           
  Misses        726      726

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@filipecosta90 filipecosta90 merged commit 078ac5d into master Jan 10, 2023
@filipecosta90 filipecosta90 deleted the perf-reporting branch January 10, 2023 16:26
rafie pushed a commit that referenced this pull request Jan 31, 2023
@filipecosta90 @rafie
Enabled Pull Request CI Performance info generation and publishing (#911
b5a3c29 
)

* Enabled Pull Request CI Performance info generation and publishing

* Updated label triggering to ensure default flow is not canceled

(cherry picked from commit 078ac5d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rafie rafie rafie approved these changes

Assignees
No one assigned
Labels
action:run-benchmark
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@filipecosta90 @rafie