Skip to content

[In-Progress] Extending ingestion benchmarks. Included a 10M docs single field benchmark #1180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

[In-Progress] Extending ingestion benchmarks. Included a 10M docs single field benchmark #1180

wants to merge 10 commits into from

Conversation

filipecosta90
Copy link
Contributor

@filipecosta90 filipecosta90 commented Feb 6, 2024 •
edited
Loading

Bellow, here's a quick sample of each added test:

  • JSON.SET of 10M docs with a single text field within an array
    sample command:
"JSON.SET" "doc:single:bb90e0afaf4b4232968016756d5755ad:0" "." "[{\"field1\": \"03247JPN5JWWG00T\"}]"
  • JSON.SET of 10M docs with a single numeric int field within an array
    sample command:
"JSON.SET" "doc:single:bb90e0afaf4b4232968016756d5755ad:0" "." "[{\"field1\": 3979443}]"
  • JSON.SET of 10M docs with a single nested array with a text field within it
    sample command:
"JSON.SET" "doc:single:a96a2f91fa0b42a18739c466cd57a633:266100" "." "[{\"field1\": [\"HNQDDNUFELI\"]}]
  • JSON.SET of 10M docs with a single nested array with a dict with text field within it
    sample command:
"JSON.SET" "doc:single:8f2d70bd81cd4b4691c60408d3dac364:849454" "." "[{\"field1\": {\"field2\": \"BM8RR8UVTS8TD\"}}]"
  • JSON.SET of 10M docs with a single nested array with a dict with an array with a text field within it
    sample command:
"JSON.SET" "doc:single:c00101ca14fd4caeace60bb74067bb3d:761505" "." "[{\"field1\": {\"field2\": [\"IV9UE15ZYUPXSX\"]}}]"
  • JSON.SET of 10M docs with a single nested array with a dict with numeric field within it
    sample command:
"JSON.SET" "doc:single:8f2d70bd81cd4b4691c60408d3dac364:849454" "." "[{\"field1\": {\"field2\": 1111}}]"
  • JSON.SET of 10M docs with a single text field
    sample command:
"JSON.SET" "doc:single:36005850af32483692828d52805ba519:0" "." "{\"field1\": \"03247JPN5JWWG00T\"}"
  • JSON.SET of 10M docs with a single numeric float field
"JSON.SET" "doc:single:56ac1c98413d4027a93f53ebf86101f5:99" "." "{\"field1\": 0.7904532831270685}"
  • JSON.SET of 10M docs with a single numeric int field
"JSON.SET" "doc:single:ec2dfc5c975d4f82887a32741f1dc32b:0" "." "{\"field1\": 3979443}"
  • JSON.SET of 10M docs with a single bool field
 "JSON.SET" "doc:single:80698057e4cd4abeb035f317607cf1af:0" "." "{\"field1\": false}"

@filipecosta90 filipecosta90 changed the title Extending ingestion benchmarks. Included a 10M docs single field benchmark [In-Progress] Extending ingestion benchmarks. Included a 10M docs single field benchmark Feb 6, 2024
@filipecosta90
Copy link
Contributor Author

filipecosta90 commented Feb 6, 2024 •
edited
Loading

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 21 stable tests between versions.
  • Detected a total of 3 highly unstable benchmarks.
  • Detected a total of 5 improvements above the improvement water line.
  • Detected a total of 12 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and extend.ingestion.benchmarks.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison extend.ingestion.benchmarks (median obs. +- std.dev) % change (higher-better) Note
json_arrappend_geojson 75455 +- 3.2% (5 datapoints) 70166 -7.0% REGRESSION
json_get_ResultSet.totalResultsAvailable_jsonsl-yahoo2_json 132440 +- 8.0% (5 datapoints) 119039 -10.1% waterline=8.0%. REGRESSION
json_get_[0]_jsonsl-1 122690 +- 6.9% (5 datapoints) 123451 0.6% waterline=6.9%. No Change
json_get_[7]_jsonsl-1 135124 +- 7.6% (5 datapoints) 112984 -16.4% waterline=7.6%. REGRESSION
json_get_[8].zero_jsonsl-1 125777 +- 4.3% (5 datapoints) 128199 1.9% No Change
json_get_[web-app].servlet[0][servlet-name]_json-parser-0000 98756 +- 5.5% (5 datapoints) 105252 6.6% waterline=5.5%. IMPROVEMENT
json_get_[web-app].servlet[0]_json-parser-0000 81623 +- 4.4% (5 datapoints) 77211 -5.4% REGRESSION
json_get_[web-app].servlet_json-parser-0000 80636 +- 6.1% (5 datapoints) 71674 -11.1% waterline=6.1%. REGRESSION
json_get_array_of_docs[1]_pass_100_json 112352 +- 4.9% (5 datapoints) 119039 6.0% IMPROVEMENT
json_get_array_of_docs[1]sclr_pass_100_json 108686 +- 2.4% (5 datapoints) 113629 4.5% potential IMPROVEMENT
json_get_array_of_docs_pass_100_json 128195 +- 6.5% (5 datapoints) 119752 -6.6% waterline=6.5%. REGRESSION
json_get_fulldoc_json-parser-0000 80961 +- 5.8% (5 datapoints) 84380 4.2% waterline=5.8%. potential IMPROVEMENT
json_get_fulldoc_jsonsl-1 101513 +- 3.6% (5 datapoints) 99000 -2.5% No Change
json_get_fulldoc_jsonsl-yahoo2_json 27201 +- 2.7% (5 datapoints) 25797 -5.2% REGRESSION
json_get_fulldoc_jsonsl-yelp_json 14175 +- 3.7% (5 datapoints) 13594 -4.1% potential REGRESSION
json_get_fulldoc_pass_100_json 125780 +- 6.6% (5 datapoints) 125777 -0.0% waterline=6.6%. No Change
json_get_key_empty 145977 +- 8.8% (5 datapoints) 132443 -9.3% waterline=8.8%. REGRESSION
json_get_message.code_jsonsl-yelp_json 121945 +- 9.6% (5 datapoints) 135124 10.8% waterline=9.6%. IMPROVEMENT
json_get_sclr_pass_100_json 138877 +- 6.4% (5 datapoints) 116271 -16.3% waterline=6.4%. REGRESSION
json_get_sub_doc.sclr_pass_100_json 121200 +- 8.5% (5 datapoints) 138881 14.6% waterline=8.5%. IMPROVEMENT
json_get_sub_doc_pass_100_json 135124 +- 8.8% (5 datapoints) 125777 -6.9% waterline=8.8%. potential REGRESSION
json_numincrby_num_1 121942 +- 6.3% (5 datapoints) 120476 -1.2% waterline=6.3%. No Change
json_nummultby_num_2 114278 +- 12.7% UNSTABLE (5 datapoints) 122690 7.4% UNSTABLE (very high variance)
json_recursive_descent_with_filter_uid_issue674 159 +- 1.0% (5 datapoints) 158 -0.6% No Change
json_set_ResultSet.totalResultsAvailable_1_jsonsl-yahoo2_json 124217 +- 4.4% (5 datapoints) 123448 -0.6% No Change
json_set_[0]foo_jsonsl-1 110487 +- 8.6% (5 datapoints) 114932 4.0% waterline=8.6%. potential IMPROVEMENT
json_set_[web-app].servlet[0][servlet-name]_bar_json-parser-0000 94778 +- 2.2% (5 datapoints) 97550 2.9% No Change
json_set_fulldoc_api_replies_q1_google_autocomplete 48473 +- 5.2% (5 datapoints) 46072 -5.0% waterline=5.2%. potential REGRESSION
json_set_fulldoc_api_replies_q2_gmaps_areatraffic 72983 +- 10.5% UNSTABLE (5 datapoints) 79356 8.7% UNSTABLE (very high variance)
json_set_fulldoc_api_replies_q3_gmaps_passiveassist 41268 +- 2.9% (5 datapoints) 41742 1.1% No Change
json_set_fulldoc_api_replies_q4_gmaps_assist 105254 +- 7.8% (5 datapoints) 91108 -13.4% waterline=7.8%. REGRESSION
json_set_fulldoc_pass-json-parser-0000 39671 +- 2.6% (5 datapoints) 38083 -4.0% potential REGRESSION
json_set_fulldoc_pass_100_json 93449 +- 5.1% (5 datapoints) 91316 -2.3% waterline=5.1%. No Change
json_set_fulldoc_yahoo2 11350 +- 1.6% (5 datapoints) 10945 -3.6% potential REGRESSION
json_set_key_empty 156240 +- 5.3% (5 datapoints) 128195 -17.9% waterline=5.3%. REGRESSION
json_set_message.code_1_jsonsl-yelp_json 116951 +- 6.1% (5 datapoints) 122690 4.9% waterline=6.1%. potential IMPROVEMENT
json_set_num_0 153322 +- 4.4% (5 datapoints) 152662 -0.4% No Change
json_set_sclr_1_pass_100_json 115599 +- 7.4% (5 datapoints) 130712 13.1% waterline=7.4%. IMPROVEMENT
json_set_sclr_pass_100_json 117639 +- 10.7% UNSTABLE (5 datapoints) 115596 -1.7% UNSTABLE (very high variance)
json_vs_hashes_hset_key_simple 154182 +- 4.7% (4 datapoints) 148690 -3.6% potential REGRESSION
json_vs_hashes_json.set_key_simple 135128 +- 5.2% (5 datapoints) 123449 -8.6% waterline=5.2%. REGRESSION

@codecov Codecov
Copy link

codecov bot commented Apr 10, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.20%. Comparing base (79c7497) to head (ee55fbc).

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #1180   +/-   ##
=======================================
  Coverage   78.20%   78.20%           
=======================================
  Files          14       14           
  Lines        3895     3895           
=======================================
  Hits         3046     3046           
  Misses        849      849

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
action:run-benchmark
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@filipecosta90