Introduction of ijson to reduce memory consumption #563

gkorland · 2021-12-06T12:04:51Z

No description provided.

* Return serde_json:Value support * Implement manager for IValue

Cargo.toml

MeirShpilraien · 2021-12-07T21:48:58Z

Fix inaccurate tests
Allow json back-end (serde_json:Value or IValue) to be configurable on startup

oshadmi

Looks great! Just small non-blocking comments

oshadmi · 2021-12-08T04:45:10Z

src/lib.rs

+    get_manage: {
+        match MANAGER {
+            ManagerType::IValue => Some(ivalue_manager::RedisIValueJsonKeyManager{phantom:PhantomData}),
+            _ => None,


Can already handle ManagerType::SerdeValue also here?

No, because then you have a match that return different types

oshadmi · 2021-12-08T06:55:04Z

src/ivalue_manager.rs

+        if path.is_empty() {
+            // update the root
+            let root = self.get_value().unwrap().unwrap();
+            let val = if root.is_object() {


Can avoid is_object with something like
if let Some(o) = root.as_object_mut()

oshadmi · 2021-12-08T07:38:14Z

src/ivalue_manager.rs

+                                v.serialize(&mut out).unwrap();
+                                self.from_str(
+                                    &String::from_utf8(out.into_inner()).unwrap(),
+                                    Format::JSON,


Not for now (there are missing implementations and tests for BSON), but maybe could be done without passing through JSON string?

We need to add support for this on ijson repository

… that can get either SERDE_JSON or IJSON, default is IJSON.

oshadmi · 2021-12-09T18:35:33Z

Fix #569

oshadmi

👍🏼

filipecosta90 · 2021-12-09T19:36:57Z

@gkorland @MeirShpilraien @oshadmi , adding a bit of context of the small regressions this PR will add to the achievable ops/sec ( at the benefit of large memory savings ).
This table has already been discussed deeply with @oshadmi and the team. The bellow tables are only focusing on achievable ops/sec and don't focus on the benefits of memory savings. ( lets say this focus only on the bad part :) )

regressions/potential regressions only:

# Comparison between master and gkorland-ijson for metric: Tests.Overall.rps. Time Period from 7 days ago to now
|                 Test Case                 |Baseline (median obs. +- std.dev)|Comparison (median obs. +- std.dev)|% change (higher-better)|           Note           |
|-------------------------------------------|---------------------------------|-----------------------------------|------------------------|--------------------------|
|json_get_[web-app].servlet_json-parser-0000| 83670.800 +- 4.2%               | 76478.290 +- 4.2%                 |-8.6%                   |REGRESSION                |
|json_get_array_of_docs_pass_100_json       | 134220.980 +- 5.4%              | 126385.935 +- 5.6%                |-5.8%                   |waterline=5.6%. REGRESSION|
|json_get_fulldoc_json-parser-0000          | 91112.125 +- 4.6%               | 79830.445 +- 0.7%                 |-12.4%                  |REGRESSION                |
|json_get_fulldoc_jsonsl-1                  | 103082.160 +- 3.5%              | 94332.820 +- 1.2%                 |-8.5%                   |REGRESSION                |
|json_get_fulldoc_jsonsl-yahoo2_json        | 28563.270 +- 4.8%               | 25967.365 +- 3.0%                 |-9.1%                   |REGRESSION                |
|json_get_fulldoc_jsonsl-yelp_json          | 14483.940 +- 4.0%               | 13019.615 +- 1.7%                 |-10.1%                  |REGRESSION                |
|json_get_key_empty                         | 159994.890 +- 4.6%              | 146046.905 +- 7.6%                |-8.7%                   |waterline=7.6%. REGRESSION|
|json_get_sub_doc_pass_100_json             | 140837.140 +- 5.3%              | 126174.675 +- 6.5%                |-10.4%                  |waterline=6.5%. REGRESSION|
|json_set_num_0                             | 153836.690 +- 5.0%              | 144919.140 +- 4.4%                |-5.8%                   |waterline=5.0%. REGRESSION|

full data:

2021-12-09 19:32:57,027 INFO Using deployment_type=oss-standalone and deployment_name=oss-standalone for the analysis
2021-12-09 19:32:57,028 INFO Using a time-delta from 7 days ago to now
2021-12-09 19:32:57,123 WARNING Based on test-cases set (key=ci.benchmarks.redislabs/circleci/RedisJSON/RedisJSON:testcases) we have 43 distinct benchmarks. 
2021-12-09 19:33:05,519 INFO Printing differential analysis between branches
# Comparison between master and gkorland-ijson for metric: Tests.Overall.rps. Time Period from 7 days ago to now
|                           Test Case                            |Baseline (median obs. +- std.dev)|Comparison (median obs. +- std.dev)|% change (higher-better)|                Note                 |
|----------------------------------------------------------------|---------------------------------|-----------------------------------|------------------------|-------------------------------------|
|json_arrappend_geojson                                          | 60141.085 +- 4.2%               | 57546.995 +- 3.6%                 |-4.3%                   |potential REGRESSION                 |
|json_get_ResultSet.totalResultsAvailable_jsonsl-yahoo2_json     | 140833.170 +- 4.3%              | 130518.540 +- 9.4%                |-7.3%                   |waterline=9.4%. potential REGRESSION |
|json_get_[0]_jsonsl-1                                           | 132443.310 +- 5.4%              | 136549.155 +- 5.0%                |3.1%                    |waterline=5.4%. potential IMPROVEMENT|
|json_get_[7]_jsonsl-1                                           | 144919.140 +- 4.1%              | 146513.525 +- 0.9%                |1.1%                    |potential IMPROVEMENT                |
|json_get_[8].zero_jsonsl-1                                      | 134220.980 +- 4.6%              | 132894.710 +- 3.4%                |-1.0%                   |potential REGRESSION                 |
|json_get_[web-app].servlet[0][servlet-name]_json-parser-0000    | 119415.315 +- 5.4%              | 117633.210 +- 3.1%                |-1.5%                   |waterline=5.4%. potential REGRESSION |
|json_get_[web-app].servlet[0]_json-parser-0000                  | 90489.550 +- 4.2%               | 88886.430 +- 3.9%                 |-1.8%                   |potential REGRESSION                 |
|json_get_[web-app].servlet_json-parser-0000                     | 83670.800 +- 4.2%               | 76478.290 +- 4.2%                 |-8.6%                   |REGRESSION                           |
|json_get_array_of_docs[1]_pass_100_json                         | 131568.560 +- 5.4%              | 125798.115 +- 1.7%                |-4.4%                   |waterline=5.4%. potential REGRESSION |
|json_get_array_of_docs[1]sclr_pass_100_json                     | 133326.220 +- 3.5%              | 130717.710 +- 1.5%                |-2.0%                   |potential REGRESSION                 |
|json_get_array_of_docs_pass_100_json                            | 134220.980 +- 5.4%              | 126385.935 +- 5.6%                |-5.8%                   |waterline=5.6%. REGRESSION           |
|json_get_fulldoc_json-parser-0000                               | 91112.125 +- 4.6%               | 79830.445 +- 0.7%                 |-12.4%                  |REGRESSION                           |
|json_get_fulldoc_jsonsl-1                                       | 103082.160 +- 3.5%              | 94332.820 +- 1.2%                 |-8.5%                   |REGRESSION                           |
|json_get_fulldoc_jsonsl-yahoo2_json                             | 28563.270 +- 4.8%               | 25967.365 +- 3.0%                 |-9.1%                   |REGRESSION                           |
|json_get_fulldoc_jsonsl-yelp_json                               | 14483.940 +- 4.0%               | 13019.615 +- 1.7%                 |-10.1%                  |REGRESSION                           |
|json_get_fulldoc_pass_100_json                                  | 135127.830 +- 4.4%              | 132017.395 +- 5.9%                |-2.3%                   |waterline=5.9%. potential REGRESSION |
|json_get_key_empty                                              | 159994.890 +- 4.6%              | 146046.905 +- 7.6%                |-8.7%                   |waterline=7.6%. REGRESSION           |
|json_get_message.code_jsonsl-yelp_json                          | 136978.800 +- 5.8%              | 136640.225 +- 4.6%                |-0.2%                   |waterline=5.8%. potential REGRESSION |
|json_get_sclr_pass_100_json                                     | 142848.980 +- 6.6%              | 149253.145 +- 5.8%                |4.5%                    |waterline=6.6%. potential IMPROVEMENT|
|json_get_sub_doc.sclr_pass_100_json                             | 136047.020 +- 4.8%              | 137947.820 +- 2.8%                |1.4%                    |potential IMPROVEMENT                |
|json_get_sub_doc_pass_100_json                                  | 140837.140 +- 5.3%              | 126174.675 +- 6.5%                |-10.4%                  |waterline=6.5%. REGRESSION           |
|json_numincrby_num_1                                            | 138881.170 +- 4.2%              | 139857.170 +- 3.1%                |0.7%                    |potential IMPROVEMENT                |
|json_nummultby_num_2                                            | 136043.330 +- 5.8%              | 140837.140 +- 2.7%                |3.5%                    |waterline=5.8%. potential IMPROVEMENT|
|json_set_ResultSet.totalResultsAvailable_1_jsonsl-yahoo2_json   | 118331.980 +- 3.9%              | 109944.655 +- 7.5%                |-7.1%                   |waterline=7.5%. potential REGRESSION |
|json_set_[0]foo_jsonsl-1                                        | 121203.310 +- 5.6%              | 124993.965 +- 5.2%                |3.1%                    |waterline=5.6%. potential IMPROVEMENT|
|json_set_[web-app].servlet[0][servlet-name]_bar_json-parser-0000| 95229.020 +- 4.0%               | 91604.275 +- 5.6%                 |-3.8%                   |waterline=5.6%. potential REGRESSION |
|json_set_fulldoc_api_replies_q1_google_autocomplete             | 41949.015 +- 28.8% UNSTABLE     | 42005.690 +- 9.9%                 |0.1%                    |UNSTABLE (very high variance)        |
|json_set_fulldoc_api_replies_q2_gmaps_areatraffic               | 76697.295 +- 29.7% UNSTABLE     | 73250.950 +- 18.9% UNSTABLE       |-4.5%                   |UNSTABLE (very high variance)        |
|json_set_fulldoc_api_replies_q3_gmaps_passiveassist             | 39545.540 +- 28.8% UNSTABLE     | 37405.970 +- 11.9% UNSTABLE       |-5.4%                   |UNSTABLE (very high variance)        |
|json_set_fulldoc_api_replies_q4_gmaps_assist                    | 96183.130 +- 28.5% UNSTABLE     | 97316.020 +- 6.8%                 |1.2%                    |UNSTABLE (very high variance)        |
|json_set_fulldoc_pass-json-parser-0000                          | 32309.590 +- 27.2% UNSTABLE     | 31236.920 +- 5.7%                 |-3.3%                   |UNSTABLE (very high variance)        |
|json_set_fulldoc_pass_100_json                                  | 90080.350 +- 9.2%               | 85151.740 +- 8.9%                 |-5.5%                   |waterline=9.2%. potential REGRESSION |
|json_set_fulldoc_yahoo2                                         | 7747.390 +- 13.7% UNSTABLE      | 7939.600 +- 2.6%                  |2.5%                    |UNSTABLE (very high variance)        |
|json_set_key_empty                                              | 157470.390 +- 4.5%              | 151514.675 +- 7.0%                |-3.8%                   |waterline=7.0%. potential REGRESSION |
|json_set_message.code_1_jsonsl-yelp_json                        | 117638.760 +- 6.2%              | 116610.910 +- 0.3%                |-0.9%                   |waterline=6.2%. potential REGRESSION |
|json_set_num_0                                                  | 153836.690 +- 5.0%              | 144919.140 +- 4.4%                |-5.8%                   |waterline=5.0%. REGRESSION           |
|json_set_sclr_1_pass_100_json                                   | 128195.270 +- 5.0%              | 122856.800 +- 5.4%                |-4.2%                   |waterline=5.4%. potential REGRESSION |
|json_set_sclr_pass_100_json                                     | 121203.310 +- 6.5%              | 116634.705 +- 6.8%                |-3.8%                   |waterline=6.8%. potential REGRESSION |
|json_vs_hashes_hset_key_simple                                  | 164600.920 +- 5.2%              | 161799.155 +- 4.7%                |-1.7%                   |waterline=5.2%. potential REGRESSION |
|json_vs_hashes_json.set_key_simple                              | 143876.610 +- 4.7%              | 144384.900 +- 5.0%                |0.4%                    |potential IMPROVEMENT                |
2021-12-09 19:33:05,622 INFO Detected a total of 25 stable tests between versions.
2021-12-09 19:33:05,622 WARNING Detected a total of 6 highly unstable benchmarks.
2021-12-09 19:33:05,622 WARNING Detected a total of 9 regressions bellow the regression water line.

* initial ijson commit * Code arrangement * Return serde_json:Value support * Implement manager for IValue * Implement lasts unimplimented manager API for IValue * fmt fixes * return jsonpath library to point to generic_json_path * Made backend configurable on start using JSON_BACKEND module argument that can get either SERDE_JSON or IJSON, default is IJSON. * run tests on both backends * json_init -> json_init_config * fmt fixes * tests fixes * fmt fixes * disable memory test as its currently incorrect * review fixes * update redismodule-rs to 0.25 * fix 6.0 tests * Skip short read test on 6.0 (too slow) * Skip module keyspace notification test (introduced in redis 6.2) Co-authored-by: meir <meir@redis.com> Co-authored-by: oshadmi <omer.shadmi@redislabs.com> (cherry picked from commit b6a070d)

Merge ijson (#563) to 2.0

* Fixes in config.yml (#409) (cherry picked from commit 8adfd29) * Updated module version * json.get returns a top-level array (as bulk string) (#411) (#412) * json.get returns a top-level array (as bulk string) * Fix format and tests (cherry picked from commit cc6d225) * Updated modules version (2) * Added integration branch 2.0 * Updated module version * Updated modules version (2) * Added integration branch 2.0 * [2.0] Updated readies * [2.0] Updated readies (for docker cpuset) * update readies to master * Update version to 2.0.2 * Update Cargo.toml version to 2.0.2 * Update deps: jsonpath tag v2.0.2 * Multipath: handle default arguments (#503) * Multipath: arrlen - handle missing path * Multipath: handle/add tests for default args (strlen objkeys objlen resp get strappend arrpop) (cherry picked from commit 8733934) * QA automation update (#505) (cherry picked from commit 223316f) * system-setup: added binutils for macOS (objcopy) (#524) (cherry picked from commit e6da70b) * [2.0] Cherry-pick #524, updated readies * Update version to 2.0.3 * Update readies * Bump version to 2.0.4 * Update commands.json file (2) (#534) (cherry picked from commit 58ea6f1) * Updated RS_VERSIONS (#536) (cherry picked from commit d7ee051) * Doc 2.0 (#538) (#540) * Document new path, JSON.GET and JSON.MGET * Add examples and update results for all commands * Update complexity information * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add example to NUMINCRBY. Fix example in NUMMULTBY * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Post review rephrasing * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add indexing JSON documents * Post review * Update docs/indexing_JSON.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add indexing json doc (#543) * Doc 2.0 (#538) * Document new path, JSON.GET and JSON.MGET * Add examples and update results for all commands * Update complexity information * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add example to NUMINCRBY. Fix example in NUMMULTBY * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Post review rephrasing * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add indexing JSON documents * Post review * Update docs/indexing_JSON.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add indexing json to the doc * update to latest readies Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> Co-authored-by: oshadmi <omer.shadmi@redislabs.com> * Build system updates (esp. target into BINDIR) (#547) (cherry picked from commit 09c4618) * [2.0] Updated readies * [2.0] Dockerfile: updated REDISEARCH_VERSION * ARM support (#550) (cherry picked from commit c3ed883) * Bump version to 2.0.5 * Build ARM for bionic; test with Redis 6.0 (#556) (#557) (cherry picked from commit b1c8cb4) * initial ijson commit (#563) * initial ijson commit * Code arrangement * Return serde_json:Value support * Implement manager for IValue * Implement lasts unimplimented manager API for IValue * fmt fixes * return jsonpath library to point to generic_json_path * Made backend configurable on start using JSON_BACKEND module argument that can get either SERDE_JSON or IJSON, default is IJSON. * run tests on both backends * json_init -> json_init_config * fmt fixes * tests fixes * fmt fixes * disable memory test as its currently incorrect * review fixes * update redismodule-rs to 0.25 * fix 6.0 tests * Skip short read test on 6.0 (too slow) * Skip module keyspace notification test (introduced in redis 6.2) Co-authored-by: meir <meir@redis.com> Co-authored-by: oshadmi <omer.shadmi@redislabs.com> (cherry picked from commit b6a070d) * path.md copyedit suggestions (cherry picked from commit 57e6a87) * indexing_JSON.md copyedit suggestions (cherry picked from commit 8cf0cd4) * Update Cargo.toml (#552) Co-authored-by: Rafi Einstein <rafi@redislabs.com> (cherry picked from commit 42e6f32) * Update freebsd.yml (#565) * Update freebsd.yml (cherry picked from commit 58d6fbf) * JSON.SET full doc benchmark extensions (#568) * JSON.SET full doc benchmark extensions * [fix] Adjusted json_vs_hashes_json.set_key_simple to use a larger keyspace range * [fix] Fixed q5 JSON.SET test (cherry picked from commit 3ec45da) * Bump mkdocs from 1.1.2 to 1.2.3 in /docs (#572) Bumps [mkdocs](https://github.com/mkdocs/mkdocs) from 1.1.2 to 1.2.3. - [Release notes](https://github.com/mkdocs/mkdocs/releases) - [Commits](mkdocs/mkdocs@1.1.2...1.2.3) --- updated-dependencies: - dependency-name: mkdocs dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 91892dd) * Add docs fo JSON.CLEAR and JSON.TOGGLE (#474) (#607) * fix #465 Add docs for JSON.CLEAR and JSON.TOGGLE * Document adding a new child with JSON.SET (#622) (#626) * Document adding a new child with JSON.SET * Restoring previous edits * Restoring previous edits (2) (cherry picked from commit cd2458d) Co-authored-by: Rafi Einstein <rafi@redislabs.com> Co-authored-by: Emmanuel Keller <74923777+emmanuelkeller@users.noreply.github.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Rachel Elledge <86307637+rrelledge@users.noreply.github.com> Co-authored-by: filipe oliveira <filipecosta.90@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Merge from master to 2.0 (towards 2.0.7) (#665) * Fixes in config.yml (#409) (cherry picked from commit 8adfd29) * Updated module version * json.get returns a top-level array (as bulk string) (#411) (#412) * json.get returns a top-level array (as bulk string) * Fix format and tests (cherry picked from commit cc6d225) * Updated modules version (2) * Added integration branch 2.0 * Updated module version * Updated modules version (2) * Added integration branch 2.0 * [2.0] Updated readies * [2.0] Updated readies (for docker cpuset) * update readies to master * Update version to 2.0.2 * Update Cargo.toml version to 2.0.2 * Update deps: jsonpath tag v2.0.2 * Multipath: handle default arguments (#503) * Multipath: arrlen - handle missing path * Multipath: handle/add tests for default args (strlen objkeys objlen resp get strappend arrpop) (cherry picked from commit 8733934) * QA automation update (#505) (cherry picked from commit 223316f) * system-setup: added binutils for macOS (objcopy) (#524) (cherry picked from commit e6da70b) * [2.0] Cherry-pick #524, updated readies * Update version to 2.0.3 * Update readies * Bump version to 2.0.4 * Update commands.json file (2) (#534) (cherry picked from commit 58ea6f1) * Updated RS_VERSIONS (#536) (cherry picked from commit d7ee051) * Doc 2.0 (#538) (#540) * Document new path, JSON.GET and JSON.MGET * Add examples and update results for all commands * Update complexity information * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add example to NUMINCRBY. Fix example in NUMMULTBY * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Post review rephrasing * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add indexing JSON documents * Post review * Update docs/indexing_JSON.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add indexing json doc (#543) * Doc 2.0 (#538) * Document new path, JSON.GET and JSON.MGET * Add examples and update results for all commands * Update complexity information * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add example to NUMINCRBY. Fix example in NUMMULTBY * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Post review rephrasing * Update docs/commands.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add indexing JSON documents * Post review * Update docs/indexing_JSON.md Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Add indexing json to the doc * update to latest readies Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> Co-authored-by: oshadmi <omer.shadmi@redislabs.com> * Build system updates (esp. target into BINDIR) (#547) (cherry picked from commit 09c4618) * [2.0] Updated readies * [2.0] Dockerfile: updated REDISEARCH_VERSION * ARM support (#550) (cherry picked from commit c3ed883) * Bump version to 2.0.5 * Build ARM for bionic; test with Redis 6.0 (#556) (#557) (cherry picked from commit b1c8cb4) * initial ijson commit (#563) * initial ijson commit * Code arrangement * Return serde_json:Value support * Implement manager for IValue * Implement lasts unimplimented manager API for IValue * fmt fixes * return jsonpath library to point to generic_json_path * Made backend configurable on start using JSON_BACKEND module argument that can get either SERDE_JSON or IJSON, default is IJSON. * run tests on both backends * json_init -> json_init_config * fmt fixes * tests fixes * fmt fixes * disable memory test as its currently incorrect * review fixes * update redismodule-rs to 0.25 * fix 6.0 tests * Skip short read test on 6.0 (too slow) * Skip module keyspace notification test (introduced in redis 6.2) Co-authored-by: meir <meir@redis.com> Co-authored-by: oshadmi <omer.shadmi@redislabs.com> (cherry picked from commit b6a070d) * path.md copyedit suggestions (cherry picked from commit 57e6a87) * indexing_JSON.md copyedit suggestions (cherry picked from commit 8cf0cd4) * Update Cargo.toml (#552) Co-authored-by: Rafi Einstein <rafi@redislabs.com> (cherry picked from commit 42e6f32) * Update freebsd.yml (#565) * Update freebsd.yml (cherry picked from commit 58d6fbf) * JSON.SET full doc benchmark extensions (#568) * JSON.SET full doc benchmark extensions * [fix] Adjusted json_vs_hashes_json.set_key_simple to use a larger keyspace range * [fix] Fixed q5 JSON.SET test (cherry picked from commit 3ec45da) * Bump mkdocs from 1.1.2 to 1.2.3 in /docs (#572) Bumps [mkdocs](https://github.com/mkdocs/mkdocs) from 1.1.2 to 1.2.3. - [Release notes](https://github.com/mkdocs/mkdocs/releases) - [Commits](mkdocs/mkdocs@1.1.2...1.2.3) --- updated-dependencies: - dependency-name: mkdocs dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 91892dd) * Add docs fo JSON.CLEAR and JSON.TOGGLE (#474) (#607) * fix #465 Add docs for JSON.CLEAR and JSON.TOGGLE * Document adding a new child with JSON.SET (#622) (#626) * Document adding a new child with JSON.SET * Restoring previous edits * Restoring previous edits (2) (cherry picked from commit cd2458d) Co-authored-by: Rafi Einstein <rafi@redislabs.com> Co-authored-by: Emmanuel Keller <74923777+emmanuelkeller@users.noreply.github.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Rachel Elledge <86307637+rrelledge@users.noreply.github.com> Co-authored-by: filipe oliveira <filipecosta.90@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * revert bad merge (#668) * revert bad merge (2) (#669) * Add test to MOD2099 + update dep to latest generic_json_path (#671) * Fox JSON.CLEAR doc * Allow QA tests on a specific RS_VERSION (#682) * Build fixes, inc. Rocky Linux 8 (#685) * Build fixes, inc. Rocky Linux 8 * Use jsonpath lib with fix for issue #667 (#691) * Upgrade to latest jsonpath lib and add test for issue 667 * Fix comment * MOD-2785: Add benchmark for issue #674 (#693) * Add benchmark for issue #674 * Fix yml * Using memtier_benchmark for more granular control on json_recursive_descent_with_filter_uid_issue674 benchmark Co-authored-by: filipecosta90 <filipecosta.90@gmail.com> * Update release.json remove xenial - not supported add rhel8 * Enabled CI profilers (#692) * Enabled CI profilers * Enabled dwarf call graph mode and fail-fast on CI perf Co-authored-by: Rafi Einstein <rafi@redislabs.com> Co-authored-by: Emmanuel Keller <74923777+emmanuelkeller@users.noreply.github.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Rachel Elledge <86307637+rrelledge@users.noreply.github.com> Co-authored-by: filipe oliveira <filipecosta.90@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: tomerhekredis <72793005+tomerhekredis@users.noreply.github.com>

gkorland and others added 4 commits December 6, 2021 13:59

initial ijson commit

f3da896

Code arrangement

932ec7e

* Return serde_json:Value support * Implement manager for IValue

Implement lasts unimplimented manager API for IValue

edd93d4

fmt fixes

6781c41

MeirShpilraien requested a review from oshadmi December 7, 2021 21:46

MeirShpilraien marked this pull request as ready for review December 7, 2021 21:46

MeirShpilraien reviewed Dec 7, 2021

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

MeirShpilraien mentioned this pull request Dec 7, 2021

remove deadcode #564

Closed

return jsonpath library to point to generic_json_path

8e48af7

oshadmi previously approved these changes Dec 8, 2021

View reviewed changes

gkorland dismissed oshadmi’s stale review via 8e48af7 December 8, 2021 08:26

MeirShpilraien added 9 commits December 8, 2021 10:54

Made backend configurable on start using JSON_BACKEND module argument…

1aba8a9

… that can get either SERDE_JSON or IJSON, default is IJSON.

run tests on both backends

055b85a

json_init -> json_init_config

cf6e5a5

Merge branch 'master' into gkorland-ijson

35331a9

fmt fixes

1fc8249

tests fixes

6449501

fmt fixes

c09bf21

disable memory test as its currently incorrect

4d667d6

review fixes

59d3cfe

MeirShpilraien requested a review from oshadmi December 9, 2021 13:34

update redismodule-rs to 0.25

c2b831e

MeirShpilraien self-requested a review December 9, 2021 15:26

fix 6.0 tests

c5314cc

oshadmi previously approved these changes Dec 9, 2021

View reviewed changes

Skip short read test on 6.0 (too slow)

a7b76e3

oshadmi dismissed their stale review via a7b76e3 December 9, 2021 17:40

Skip module keyspace notification test (introduced in redis 6.2)

a338d20

oshadmi requested review from oshadmi December 9, 2021 18:40

oshadmi approved these changes Dec 9, 2021

View reviewed changes

MeirShpilraien merged commit b6a070d into master Dec 9, 2021

MeirShpilraien deleted the gkorland-ijson branch December 9, 2021 19:43

rafie added a commit that referenced this pull request Dec 15, 2021

Merge pull request #576 from RedisJSON/omer_merge_ijson

db3329c

Merge ijson (#563) to 2.0

K-Jo changed the title ~~initial ijson commit~~ Introduction of ijson to reduce memory consumption Dec 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduction of ijson to reduce memory consumption #563

Introduction of ijson to reduce memory consumption #563

gkorland commented Dec 6, 2021

MeirShpilraien commented Dec 7, 2021 •

edited

Loading

oshadmi left a comment

oshadmi Dec 8, 2021

MeirShpilraien Dec 8, 2021

oshadmi Dec 8, 2021

MeirShpilraien Dec 9, 2021

oshadmi Dec 8, 2021

MeirShpilraien Dec 8, 2021

oshadmi commented Dec 9, 2021

oshadmi left a comment

filipecosta90 commented Dec 9, 2021

Introduction of ijson to reduce memory consumption #563

Introduction of ijson to reduce memory consumption #563

Conversation

gkorland commented Dec 6, 2021

MeirShpilraien commented Dec 7, 2021 • edited Loading

oshadmi left a comment

Choose a reason for hiding this comment

oshadmi Dec 8, 2021

Choose a reason for hiding this comment

MeirShpilraien Dec 8, 2021

Choose a reason for hiding this comment

oshadmi Dec 8, 2021

Choose a reason for hiding this comment

MeirShpilraien Dec 9, 2021

Choose a reason for hiding this comment

oshadmi Dec 8, 2021

Choose a reason for hiding this comment

MeirShpilraien Dec 8, 2021

Choose a reason for hiding this comment

oshadmi commented Dec 9, 2021

oshadmi left a comment

Choose a reason for hiding this comment

filipecosta90 commented Dec 9, 2021

MeirShpilraien commented Dec 7, 2021 •

edited

Loading