Skip to content

[Scheduler] Continue with stored Checkpoint::$time on lock #52039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 17, 2023

Conversation

Jeroeny
Copy link
Contributor

@Jeroeny Jeroeny commented Oct 13, 2023

Q A
Branch? 6.4
Bug fix? yes
New feature? no
Deprecations? no
License MIT

When there are multiple message-consumer processes running, the Lock mechanism can be used to ensure a single process takes responsibility of the scheduling.
However when the responsible process releases the lock and a new process acquires it, the state of the checkpoint is reset on purpose.

I'm not sure what the reason behind this is. Maybe I'm missing something and this PR isn't valid. But it seems strange that a Checkpoint stores it's progress which is then lost when that process stops.

Perhaps the 'reset' was more intended like a `reload', since the cached value may have changed by another process? Then this PR would fix it to be an actual reload.

Also I think the change in logic requires this to go to 6.4 branch?

@Jeroeny Jeroeny requested a review from kbond as a code owner October 13, 2023 09:48
@carsonbot carsonbot added this to the 6.4 milestone Oct 13, 2023
@Jeroeny Jeroeny changed the title [Scheduler] Continue with checkpoint time on lock [Scheduler] Continue with stored Checkpoint::$time on lock Oct 13, 2023
@fabpot fabpot added the ❄️ Feature Freeze Important Pull Requests to finish before the next Symfony "feature freeze" label Oct 20, 2023
@nicolas-grekas nicolas-grekas removed the ❄️ Feature Freeze Important Pull Requests to finish before the next Symfony "feature freeze" label Oct 20, 2023
@nicolas-grekas
Copy link
Member

nicolas-grekas commented Nov 15, 2023

I'm trying to understand what you mean here, I'm not deep enough into the topic 😬

Does your description also apply when there is no cache defined?

How about this patch instead?

--- a/src/Symfony/Component/Scheduler/Generator/Checkpoint.php
+++ b/src/Symfony/Component/Scheduler/Generator/Checkpoint.php
@@ -37,14 +37,12 @@ final class Checkpoint implements CheckpointInterface
             return false;
         }
 
-        if ($this->reset) {
-            $this->reset = false;
-            $this->save($now, -1);
-        }
-
         if ($this->cache) {
             [$this->time, $this->index, $this->from] = $this->cache->get($this->name, fn () => [$now, -1, $now]) + [2 => $now];
             $this->save($this->time, $this->index);
+        } elseif ($this->reset) {
+            $this->reset = false;
+            $this->save($now, -1);
         }

@nicolas-grekas nicolas-grekas modified the milestones: 6.4, 7.1 Nov 15, 2023
@yceruto yceruto modified the milestones: 6.4, 7.1 Nov 15, 2023
@Jeroeny
Copy link
Contributor Author

Jeroeny commented Nov 16, 2023

Does you description also apply when there is no cache defined?

That's an interesting case. I think that may indeed be the thought behind this reset.
Because if you have two processes running without shared cache, apparently you do not want them to continue from exactly the last saved time. So maybe to prevent re-processing a certain time frame in a second worker when it has acquired a lock after another worker, it gets reset.

But, if they do have a shared cache, I think worker two can continue where worker one left off. As I'd say that is the purpose of the cache.

How about this patch instead?

With the above reasoning, that patch does indeed work. I've added a test to specifically test a second checkpoint acquiring the lock after the first and test that it continues from the saved time.

@nicolas-grekas nicolas-grekas changed the base branch from 7.1 to 6.4 November 17, 2023 21:07
@nicolas-grekas nicolas-grekas changed the base branch from 7.1 to 6.4 November 17, 2023 21:09
@nicolas-grekas
Copy link
Member

Thank you @Jeroeny.

@nicolas-grekas nicolas-grekas merged commit 6f63e25 into symfony:6.4 Nov 17, 2023
This was referenced Nov 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants