Effects #2826

swilly22 · 2023-01-14T18:59:00Z

Support for GRAPH.EFFECT replicating modifications to both replicas and AOF

codecov · 2023-02-08T08:02:16Z

Codecov Report

Patch coverage: 95.57% and project coverage change: +0.11 🎉

Comparison is base (ab4d85b) 90.36% compared to head (bebf1e7) 90.48%.

❗ Current head bebf1e7 differs from pull request most recent head b52b41f. Consider uploading reports for the commit b52b41f to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2826      +/-   ##
==========================================
+ Coverage   90.36%   90.48%   +0.11%     
==========================================
  Files         283      287       +4     
  Lines       28072    28747     +675     
==========================================
+ Hits        25367    26011     +644     
- Misses       2705     2736      +31

Impacted Files	Coverage Δ
src/commands/commands.c	`46.15% <0.00%> (-3.85%)`	⬇️
src/graph/entities/edge.c	`83.87% <ø> (ø)`
...ializers/decoders/prev/v11/decode_graph_entities.c	`0.00% <0.00%> (ø)`
...ializers/decoders/prev/v12/decode_graph_entities.c	`0.00% <0.00%> (ø)`
src/commands/cmd_config.c	`92.20% <70.00%> (-3.74%)`	⬇️
src/value.c	`91.37% <93.33%> (+0.57%)`	⬆️
src/commands/cmd_effect.c	`93.75% <93.75%> (ø)`
src/graph/graph_statistics.c	`96.15% <93.75%> (-3.85%)`	⬇️
src/undo_log/undo_log.c	`98.62% <95.00%> (+0.04%)`	⬆️
src/configuration/config.c	`90.22% <95.45%> (+0.29%)`	⬆️
... and 20 more

... and 5 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

AviAvni

nice

AviAvni · 2023-02-08T08:16:35Z

src/commands/cmd_query.c

+	UndoLog *undolog = QueryCtx_GetUndoLog();
+	uint n = UndoLog_Length(*undolog);
+
+	if(n > 2048 || n == 0) {


AviAvni · 2023-02-08T08:21:09Z

src/commands/cmd_query.c

+
+	if(n > 2048 || n == 0) {
+		// either no changes or too many changes, do not use effects
+		return false;


we don't check if there is non-deterministic change

AviAvni · 2023-02-08T08:24:40Z

src/configuration/config.h

@@ -47,11 +48,13 @@ static const Config_Option_Field RUNTIME_CONFIGS[] = {
 	Config_MAX_QUEUED_QUERIES,
 	Config_QUERY_MEM_CAPACITY,
 	Config_VKEY_MAX_ENTITY_COUNT,
-	Config_DELTA_MAX_PENDING_CHANGES
+	Config_DELTA_MAX_PENDING_CHANGES,
+	Config_REPLICATE_EFFECTS


are you sure we want it to changed during runtime?
did you test what happen if you change this during traffic?
also need to test that we can load aof file mixed with effect and queries

AviAvni · 2023-02-08T09:02:28Z

src/graph/graph_hub.c

+	bool has_indecise = GraphContext_HasIndices(gc);
+	UndoLog *undo_log = (log == true) ? QueryCtx_GetUndoLog() : NULL;
+
+	for (uint i = 0; i < n; i++) {


maybe

if(log == true) { for (uint i = 0; i < n; i++) { // add each deleted edge to undo-log UndoLog_DeleteEdge(undo_log, edges + i); } } if(has_indecise == true) { for (uint i = 0; i < n; i++) { // remove edge from relevant indices _DeleteEdgeFromIndices(gc, edges + i); } }

I don't want to perform 2N operations

AviAvni · 2023-03-02T12:41:40Z

we need to add summary of the changes and then update matrix size so the matrix policy will be not or only resize when applying effect

LiorKogan · 2023-04-06T16:03:09Z

docs/docs/configuration.md

+total execution time / number of changes:  5ms / 5 = 1ms.
+if the average modification time is greater then `EFFECTS_THRESHOLD` the query
+will be replicated to both replicas and AOF as a graph effect otherwise the original
+query will be replicated.


Suggestion: Add:
0: always replicate the effect; -1: always replicate the original query.

Please add:
(Since RedisGraph v2.12)

* update with master * wait for replica * wait after each query * refactor test * effect configuration * wip * determine when to use effects * remove stats * skip replica config * adjust constraint test * use readonly query whenever possible * debuging * pass array and len to update labels * set edge endpoints * mostly repositioning * remove debugging prints

* Effects (#2826) * update with master * wait for replica * wait after each query * refactor test * effect configuration * wip * determine when to use effects * remove stats * skip replica config * adjust constraint test * use readonly query whenever possible * debuging * pass array and len to update labels * set edge endpoints * mostly repositioning * remove debugging prints * Update graph.query.md (#3029) BFS: 3 arguments instead of 4 * Clear errno before strtol() (#3021) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Validate with fix (#3018) * wip * fix validations * touchup * no if branch when children are not visited * migrate and add tests, fix RETURN validation * doc addition --------- Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Refactor execution-plan freeing mechanism (#2946) * Refactor execution-plan freeing mechanism Co-authored-by: razmon <razmonsonego2@gmail.com> * use rax to store labels * fix comments * add comment * addressing review comments * addressed review comments * fix * leak fix * free ops deepest first * addressing review comments * Update execution_plan.c remove dict init * Update execution_plan.c include setjmp * touchup * touchup --------- Co-authored-by: Ofir Moskovich <ofir.moskovich@redislabs.com> Co-authored-by: Avi Avni <avi.avni@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix optimize label scan (2) (#3034) * introduce swap candidates to Label-Scan * doc-fix * fix * tests fix * Update test_optimizations_plan.py * Update scan_functions.h * move context to heap * fix leak * for index scan as well * addressing review comments * touchup * Update scan_functions.c * Update scan_functions.c --------- Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix LockWrite crash (#3039) * JoinConsume(): Propagaget reset if stream depleted * WIP: Test FOREACH * fix foreach crash * fix tests * fix --------- Co-authored-by: Avi Avni <avi.avni@gmail.com> * Fix for Xenial (OpenMP 4.0/4.5) (#3017) * Fix for Xenial (OpenMP 4.5) * fixes 1 * fixes 2 * fixes 3 * fixes 4 * fixes 5 * fixes 6 * fixes 7 --------- Co-authored-by: Avi Avni <avi.avni@gmail.com> * accumulate updates (#3005) * accumulate updates * fix leaks * add test * unify default dictType definitions * fix * fix 2 * add test * wip fix * wip fix * fix crash * add effect log * early review * fix build * fix test * fixes * fix * refactoring in progress * refactoring continue * write effects directly into buffer * effects-buffer * no need to clone value * no need to clone string * address review comments * address review * fix * address review comment * address reivew * fix perf issue * fix build * refactoring * fix perf * fix build * fix * fix leak * remove redundant flags * compute change against new attribute-set * fix perf * address review * fix * fix * adding comments --------- Co-authored-by: razmon <razmonsonego2@gmail.com> Co-authored-by: Raz Monsonego <74051729+raz-mon@users.noreply.github.com> Co-authored-by: swilly22 <roi@redislabs.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix rewrite `DELETE` clauses (#3067) * fix and test * add example --------- Co-authored-by: Avi Avni <avi.avni@gmail.com> * Reattach on reset (#3070) * formatting * reattach iterator on reset * disable graphblas memory pool (#3068) * bump version 2.12.1 --------- Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Lior Kogan <koganlior1@gmail.com> Co-authored-by: nafraf <nafraf@users.noreply.github.com> Co-authored-by: Raz Monsonego <74051729+raz-mon@users.noreply.github.com> Co-authored-by: Ofir Moskovich <ofir.moskovich@redislabs.com> Co-authored-by: Rafi Einstein <raffapen@outlook.com> Co-authored-by: razmon <razmonsonego2@gmail.com> Co-authored-by: swilly22 <roi@redislabs.com>

swilly22 added the enhancement label Jan 14, 2023

swilly22 requested a review from AviAvni January 14, 2023 18:59

swilly22 assigned swilly22 and raz-mon Jan 14, 2023

swilly22 force-pushed the effects branch 3 times, most recently from dd92c2b to a8c0587 Compare February 3, 2023 17:48

swilly22 added the x:perf Performance related issue/PR ( not affecting functionality ) label Feb 8, 2023

AviAvni suggested changes Feb 8, 2023

View reviewed changes

swilly22 force-pushed the effects branch 3 times, most recently from a21b477 to db97417 Compare March 30, 2023 06:02

swilly22 added 7 commits March 31, 2023 10:37

update with master

a464c40

wait for replica

ede1ef4

wait after each query

fc52ffa

refactor test

21cf618

effect configuration

3673ff4

wip

05bafef

determine when to use effects

64d0cd7

swilly22 force-pushed the effects branch from d37bf09 to 64d0cd7 Compare March 31, 2023 08:07

swilly22 added 2 commits April 2, 2023 21:10

remove stats

16b76ee

skip replica config

95b59a6

swilly22 force-pushed the effects branch from eca7f96 to 95b59a6 Compare April 3, 2023 06:14

swilly22 and others added 5 commits April 3, 2023 09:15

Merge branch 'master' into effects

56c76e8

adjust constraint test

a550916

use readonly query whenever possible

9859ef1

debuging

9b79132

pass array and len to update labels

3bfa971

swilly22 added 3 commits April 6, 2023 13:26

set edge endpoints

bebf1e7

mostly repositioning

b52b41f

remove debugging prints

f045a18

swilly22 merged commit 728d93d into master Apr 6, 2023

swilly22 deleted the effects branch April 6, 2023 15:35

LiorKogan reviewed Apr 6, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Effects #2826

Effects #2826

Uh oh!

swilly22 commented Jan 14, 2023

Uh oh!

codecov bot commented Feb 8, 2023 •

edited

Loading

Uh oh!

AviAvni left a comment

Uh oh!

AviAvni Feb 8, 2023

Uh oh!

AviAvni Feb 8, 2023

Uh oh!

AviAvni Feb 8, 2023

Uh oh!

AviAvni Feb 8, 2023

Uh oh!

swilly22 Mar 29, 2023

Uh oh!

AviAvni commented Mar 2, 2023

Uh oh!

LiorKogan Apr 6, 2023

Uh oh!

LiorKogan Apr 6, 2023

Uh oh!

Uh oh!

Effects #2826

Effects #2826

Uh oh!

Conversation

swilly22 commented Jan 14, 2023

Uh oh!

codecov bot commented Feb 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

AviAvni left a comment

Choose a reason for hiding this comment

Uh oh!

AviAvni Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

AviAvni Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

AviAvni Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

AviAvni Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

swilly22 Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

AviAvni commented Mar 2, 2023

Uh oh!

LiorKogan Apr 6, 2023

Choose a reason for hiding this comment

Uh oh!

LiorKogan Apr 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Feb 8, 2023 •

edited

Loading