[Scheduler] [Component] Scheduler #35315

Guikingone · 2020-01-12T17:58:43Z

Q	A
Branch?	master
Bug fix?	no
New feature?	yes
Deprecations?	no
Tickets	Implement #36821
License	MIT
Doc PR	To define

Hi everyone 👋

New year, new idea, first, let me explain why I open this PR as a draft, this new "experimentation" is mostly a draft as the framework integration is not completely done (I've some work pushed on a private repository).

So, what is it?

First, it's a new component, inspired by what Laravel can do with Illuminate/Scheduling but more important, it's inspired by modern platform that allows us to deploy our apps, in most platform, we can define tasks (cron and so on) and if we deploy our apps to "old" platform, we need to edit a crontab file where we need to declare all of our tasks before executing them (and waiting to fail?), the Scheduler component is here to solve most of the problems that can comes with scheduled tasks:

What if my platform does not support cron?
What if I need to plan tasks using Http?
What if I need to store tasks in a queue?
Etc...

So, what's inside?

First, we've a Scheduler, that's the entrypoint when it comes to scheduling a Task (this one can be a Http one, Cron, command, queue and so on), once scheduled, a Worker handle a task and execute it.

For now, this experimentation is not complete (the Worker need to be improved, the Messenger "bridge" also and so on) and this PR lack what I'm working on when it comes to the framework integration, this PR is opened in order to discuss and see if it could be a good idea to work on or if it's preferable to stop it now 🙂

Thanks for the feedback 👋

PS: I know that SymfonyCloud use YAML to declare tasks but we don't always use it and the Scheduler idea is here to give a "standard" solution.

Here's a small example:

Standalone

<?php 

use ...; 

$scheduler = new Scheduler('Europe/Paris');

$tasks = new ShellTask('app.test', 'echo Symfony', '*/5 * * * *');
$scheduler->schedule($task);

Framework

framework:
  scheduler:
    timezone: 'Europe/Paris'
    transports:
      local:
        dsn: 'local://'
    schedulers:
      foo:
        transport: 'local'
    tasks:
      app.foo:
        scheduler: 'foo'
        type: 'shell'
        expression: '* * * * *'
        options:
          command: 'echo Symfony!'

Roadmap

Features	State
Being able to schedule Symfony commands using the Command FQCN	✅
Adding shortcuts to create "default" cron expressions	✅
Add methods that can trigger some logic before/after the execution of the task	✅
Being able to export the scheduled tasks using txt and json formats	✅
Use transports to ease the configuration	✅
Being able to create cron files	✅
Schedule tasks for specific duration	✅
Schedule tasks for specific execution date	➡️
Adding task (and pool) priority (Pools removed)	✅
Track and optimise tasks execution	➡️
Allow to trigger tasks using a specific path (like fragments)	➡️
Scheduler factory	✅
Cloud platform bridges	➡️
FrameworkBundle integration	➡️
WebProfilerBundle integration	➡️
Notifier integration	➡️
Documentation	TODO

Core features:

Transports (under active work since January)

Commands

bin/console scheduler:consume foo, bar --limit 10 --time-limit 10
bin/console scheduler:export foo --directory /srv/app --format json --filename foo --limit 10
bin/console scheduler:generate foo, bar --directory var/schedulers/export
bin/console scheduler:get-tasks foo --expression * * * * * --state paused
bin/console scheduler:list-failed
bin/console scheduler:reboot foo --retry 10
bin/console scheduler:run foo --name app.foo --expression * * * * *

Kernel & classes as Scheduler "entrypoints" (Added on 18/05/2020)

Scheduler defines a SchedulerAwareInterface which allow virtually any class to be a entrypoint to define new tasks:

<?php

/*
 * This file is part of the Symfony package.
 *
 * (c) Fabien Potencier <fabien@symfony.com>
 *
 * For the full copyright and license information, please view the LICENSE
 * file that was distributed with this source code.
 */

namespace Symfony\Component\Scheduler;

/**
 * @author Guillaume Loulier <contact@guillaumeloulier.fr>
 */
interface SchedulerAwareInterface
{
    public function schedule(SchedulerRegistryInterface $registry): void;
}

This interface is auto configured and every "class" that implement in will receive the SchedulerRegistry which contain every declared scheduler.

Worker subscribers | Scheduler subscribers (Added on 21/05/2020)

SchedulerSubscriber are specific subscribers which are registered on every|specific scheduler(s), the registration is done thanks to getSubscribedSchedulers().

final class FooSchedulerSubscriber implements SchedulerSubscriberInterface
{
    /**
     * {@inheritdoc}
     */
    public static function getSubscribedEvents()
    {
        return ...;
    }

    /**
     * {@inheritdoc}
     */
    public static function getSubscribedSchedulers(): array
    {
        return ['*']; // But can also be ['foo', 'bar']
    }

    // ...
}

stephane-lou · 2020-01-13T07:35:15Z

I actually made a private bundle that does this. It doesn't use Messenger (for now) but allows me to schedule symfony commands using a graphical interface.
Currently I made a main command (taskamanager:execute) that checks every minute if there are scheduler commands to execute.

I also think this would be a great new component !

Guikingone · 2020-01-13T07:48:29Z

Hi @stephane-lou 👋

For now, the Messenger implementation is mainly here in order to allow to send tasks to queue and build something similar to what Notifier does, lately, I've planned to build something that allows us to do the following process:

Send a task to RabbitMQ queue
Consume the message thanks to a handler when the task expression is due
If the message task is successful, send a notification (think Slack and so on)

For the graphical interface, I think it could be a great idea (using Security of course 😉 ), this way, we can easily gave the access to this interface to a specific team department that allows them to create task without specific knowledge of the internal logic 🙂

kbond · 2020-01-13T17:32:26Z

Wow, what timing, I just open sourced a bundle for this: https://github.com/kbond/schedule-bundle

Preparing for a 1.0 release this week.

I'd be happy to help contribute if this is something that is decided should be in core.

Guikingone · 2020-01-18T08:45:24Z

Oups, wrong click, sorry for the ping FabPot :(

fabpot · 2020-01-18T10:00:56Z

The ping is actually interesting... as I do have a "Cron" component locally. It was supposed to be announced at a Symfony conference last year, but I decided to release the Notifier one first. Might come soon though.

monteiro · 2020-01-19T09:06:53Z

Thanks a lot for contributing @Guikingone.

Besides handling crons, I think this could also be interesting for one time tasks. For example, after 3 hours on a specific action, send an e-mail to try to convert a the customer instead of using a CRON.

Guikingone · 2020-01-19T10:15:02Z

@monteiro Actually, that's a planned feature, I've also planned to being able to create a pool of tasks (think about user specific tasks) that can be executed anytime 🙂

Guikingone · 2020-01-23T10:28:39Z

Hi everyone 👋

Here's a small list of recently added features (that need to be completely tested):

Task pools
Task pools & tasks priority
Failed tasks list (the Worker will retrieve it when a runner throw an Error).

Guikingone · 2020-01-23T18:19:05Z

@monteiro Regarding your feedback, I've pushed a refactor that allow to pass a valid expression handled by strtotime or even a \DatetimeInterface, the actual CRON lib allow to calculate the due date depending on a valid expression or a \DatetimeInterface

jderusse

A common issue with scheduler in a cluster, is asserting that the task will be schedule. And be scheduled once.

IMHO the component should at least:

store the last scheduled time in order to re-run missed schedules
provide a mechanism to avoid scheduling the same task twice.

jderusse · 2020-01-31T12:30:27Z

src/Symfony/Component/Scheduler/Messenger/TaskMessageHandler.php

+    {
+        $task = $message->getTask();
+
+        if (!CronExpression::factory($task->get('expression'))->isDue(new \DateTimeImmutable())) {


What if the messages took 1 hours to be proceed? Maybe isDue should take the date of creation of the task instead of now

jderusse · 2020-01-31T12:36:44Z

src/Symfony/Component/Scheduler/Messenger/TaskMessageHandler.php

+    {
+        $task = $message->getTask();
+
+        if (!CronExpression::factory($task->get('expression'))->isDue(new \DateTimeImmutable())) {


Shouldn't you take timezone into account?

src/Symfony/Component/Scheduler/Scheduler.php

Guikingone · 2020-01-31T13:00:57Z

@jderusse For the first point, that's actually the case, every time a task is runned, the subscriber define a new 'last execution date":

public function onTaskExecuted(TaskExecutedEvent $event): void
{
    $task = $event->getTask();

    $task->set('last_execution', new \DateTimeImmutable());
}

For the second point, that's an interesting point that I've not worked on directly, for now, a task cannot have the same name as an existing one, I'm gonna invest on this point 🙂

bigfoot90 · 2020-01-31T14:45:51Z

For tasks that should be executed once per time, maybe an integration with the Lock Component can be interesting.
Locks can be defined as *.pid file (locally per server node) or in a shared database/service like mysql, memcache or redis (globally per application)

Guikingone · 2020-01-31T15:05:39Z

@bigfoot90 I'm not sure that adding a new dependency (even if it's a Symfony one) its a good idea, I'm pretty sure we can find a solution without using Lock.

bigfoot90 · 2020-01-31T15:38:22Z

It should not be a hard dependecy, but a soft dependency (mentioned in suggested inside composer.json)
Then a small How to implement locking tasks guide that says:
Run composer require symfony/lock implement LockingTaskInterface in your Task class.

Guikingone · 2020-02-02T18:09:37Z

Hi everyone 👋

First, I want to thanks everyone for the first feedbacks and @nicolas-grekas for the mention on the Sf meetup slides 😊

I've pushed a refactoring which remove the notion of TaskPool, the TaskList already play this role, second, I've started an integration of the Lock component (as mentioned by @bigfoot90 and @jderusse), It's not fully in place but I'm gonna improve the tests and continue to test the integration.

Third point, I've push a first functional implementation of the Exporter using the Serializer (for formats like XML and JSON), I've planned to take a moment to look at a CLI "formatter" but I'm not sure if it worth the works.

Fourth, what's the plan now?

As I've a first functional version of the component, I'm gonna polish the tests, finish every case that I can see by far, continue on the framework integration and improve the DX and so on 🙂

Guikingone · 2020-02-03T11:26:10Z

Hi everyone,

I've just added a roadmap in the first post, the idea is to list the features that I've planned to add, which are done and so on.

Guikingone · 2020-02-03T18:33:44Z

Ok, new day, new feature, I've just pushed an ExpressionFactory which is capable of building CRON expressions using shortcuts, as I don't know if the FabPot "CRON component" will support both at and batch syntax, I don't have shortcuts for it.

Guikingone · 2020-02-04T17:42:23Z

Hi everyone 👋

New day, new feature (again, I know 😄 ), I've focused on bringing a DataCollector, for now, it collect the data during the request using a TraceableScheduler (I've planned to do the same thing for Worker), I'm not sure about what to collect by far so I've built a simple array which contains the core data linked to a task.

I've also worked on a core feature of the component, being able to execute tasks using a specific route (like fragments), I'm not sure about the implementation but definitively a fun moment playing with this.

Last but not least, I've improved the output handling when a runner finish to run a task, by default, the runner can return a string or null, the output is also send to an event.

Don't hesitate to give feedbacks 🙂

Guikingone · 2020-02-06T17:56:34Z

Hi everyone 👋

New day, new feature, I've focused the work on the DX, the Scheduler has now an interface which define core methods (resume, pause, etc).

Most of the logic is tested and in place, I've started 3 bridges for the following platforms:

Google Cloud Platform
Amazon Web Services
Microsoft Azure

I've also planned to build a bridge for SymfonyCloud but I don't know if there's a public API/SDK which allows us to plan cron tasks, for now and if it's not possible, this bridge will stay as a "planned one", for the bridges, the Google one is prettymuch in place, I've used the HttpClient component to call the REST API, the Google approach is pretty simple when it comes to scheduling tasks, Microsoft is a bit harder (and I don't talk about Amazon for now ... 😄 ).

That's all for today, don't hesitate to give feedbacks 😉

src/Symfony/Component/Scheduler/Command/ExportTaskCommand.php

src/Symfony/Component/Scheduler/Command/RunScheduledTaskCommand.php

src/Symfony/Component/Scheduler/Export/FormatterInterface.php

94noni · 2020-01-28T21:48:50Z

src/Symfony/Component/Scheduler/Messenger/TaskMessage.php

+
+    public function __construct(TaskInterface $task)
+    {
+        $this->task = $task;


Why passing the whole task and not a guid
And then the worker « restore » the task from its storage via this identifier?

src/Symfony/Component/Scheduler/README.md

94noni · 2020-01-28T21:52:35Z

src/Symfony/Component/Scheduler/Runner/RunnerInterface.php

+ */
+interface RunnerInterface
+{
+    public function run(TaskInterface $task): string;


Maybe describe the string that must be returned

For the support: bool it is obvious, but not the run: string

src/Symfony/Component/Scheduler/Task/TaskPoolInterface.php

src/Symfony/Component/Scheduler/Task/AbstractTask.php

Guikingone · 2020-02-16T18:41:46Z

Hi everyone 👋

Small update about this PR, I've pushed a "first valid implementation" of the framework configuration (not handled via the container for now) in order to demonstrate how to configure it, some parts are missing (but pushed on secondary repo).

I've also improved the syntax, some classes and the internal logic of the component, feel free to give feedbacks 🙂

Guikingone · 2020-02-17T18:44:28Z

Hi everyone 👋

Small update, I've pushed a first valid configuration along with a first YAML / XML / PHP configuration files in order to validate the current implementation, I'm gonna continue to polish the DX and the configuration in order to finish the "framework" integration.

Small ping to @fabpot (sorry, I really don't want to bother you with stupid questions 😅) in order to know if it worth to continue to work on this PR or if its a better idea to wait for your "cron" component (even if it's planned for 5.2/5.3), the idea behind this ping is really to close this PR if its a bad idea to have 2 "WIP" implementations (I'm pretty sure that yours is probably way more advanced than mine 😄).

For now, the planning of this PR is to have a "valid" and "ready to be merged" implementation for 5.2 (5.1 is too close IMO) but it all depends on the validity of working on it 🙂 .

Again, sorry for the ping and have a great evening 🙂

PS: I've use this ping because I don't know if there's a "Symfony Core-Team" or "Symfony deciders" one 😅

Guikingone · 2020-02-18T17:52:13Z

Hi everyone 👋

Small update, I've pushed a "final" version of the configuration, both YAML & PHP format are supported, the XML one continues to fail (I don't have the reason now).

The FrameworkBundle integration is almost done (same for the container injection)

Guikingone · 2020-08-11T17:23:26Z

PR rebased and ready for reviews

composer.json

src/Symfony/Component/Scheduler/Bag/BagRegistry.php

src/Symfony/Component/Scheduler/Bag/MailerBag.php

…tion

94noni

First small round :)

src/Symfony/Bundle/FrameworkBundle/Resources/config/scheduler.xml

src/Symfony/Component/Scheduler/Worker/WorkerInterface.php

src/Symfony/Component/Scheduler/Event/TaskToExecuteEvent.php

src/Symfony/Component/Scheduler/Event/WorkerStartedEvent.php

src/Symfony/Component/Scheduler/Bag/BagRegistry.php

94noni · 2020-08-12T11:52:02Z

src/Symfony/Component/Scheduler/Task/Output.php

+        $this->task = $task;
+        $this->exitCode = $exitCode;
+        $this->output = $output;
+        $this->type = $type;


is this property used somewhere?

src/Symfony/Component/Scheduler/Messenger/TaskMessageHandler.php

src/Symfony/Component/Scheduler/Export/ExporterInterface.php

94noni · 2020-08-12T12:33:02Z

src/Symfony/Component/Scheduler/EventListener/TaskSubscriber.php

+ * file that was distributed with this source code.
+ */
+
+namespace Symfony\Component\Scheduler\EventListener;


why not inside a Scheduler/EventSubscriber folder as its a Subscriber?

94noni · 2020-08-12T12:35:07Z

src/Symfony/Component/Scheduler/EventListener/StopWorkerOnTimeLimitSubscriber.php

+
+    public function onWorkerStarted(WorkerStartedEvent $event): void
+    {
+


did you forgot to call the private log function on both methods?

94noni

Second round :)

94noni · 2020-08-12T13:21:06Z

src/Symfony/Component/Scheduler/DataCollector/SchedulerDataCollector.php

+        return $this->data['schedulers'];
+    }
+
+    public function getTransports(): array


you seem not displaying it in the profiler

94noni · 2020-08-12T13:26:14Z

src/Symfony/Component/Scheduler/Task/AbstractTask.php

+            return $priority <= 1000 && $priority >= -1000;
+        });
+
+        $resolver->setInfo('arrival_time', 'The time when the task is retrieved in order to execute it');


arrival seems weird :s
if the doc says "task is retreived", why not "retreival_time" ?
just asking/challenging :)

94noni · 2020-08-12T14:50:25Z

src/Symfony/Component/Scheduler/Worker/Worker.php

+
+                try {
+                    if ($lockedTask->acquire() && !$this->isRunning()) {
+                        $this->running = true;


imo, if the method is called execute() and not run()
this should be called executing and not running (and isRunning as well)
wdyt?

src/Symfony/Component/Scheduler/Worker/Worker.php

94noni · 2020-08-12T14:55:20Z

src/Symfony/Component/Scheduler/Worker/Worker.php

+                    $this->dispatch(new WorkerStoppedEvent($this));
+
+                    return;
+                }


cant you use a finally block here?
for the process of "lock releasing / memory running false / worker stop even dispatching"

94noni · 2020-08-12T15:55:06Z

src/Symfony/Component/Scheduler/Command/ListFailedTasksCommand.php

+        $failedTasks = [];
+
+        $table = new Table($output);
+        $table->setHeaders(['Task', 'Reason', 'Date']);


i would put Date first
as date values will always have same length, but reason may not, so that the table is more clearer to read

94noni · 2020-08-12T15:56:02Z

src/Symfony/Component/Scheduler/Command/ListFailedTasksCommand.php

+        }
+
+        foreach ($failedTasksList as $task) {
+            $failedTasks[] = [$task->getName(), $task->getReason(), $task->getTriggerDate()->format('m-j-Y H:i:s')];


can you explain the choice of this date format :) ?

src/Symfony/Component/Scheduler/Command/GenerateCronCommand.php

src/Symfony/Component/Scheduler/Bridge/Redis/Transport/Connection.php

94noni · 2020-08-12T16:08:46Z

src/Symfony/Component/Scheduler/Worker/Worker.php

+    {
+        $this->dispatch(new TaskToExecuteEvent($task));
+        $output = $runner->run($task);
+        $task->set('last_execution', new \DateTimeImmutable());


should this be inside a subscriber? or it is here on purposes

94noni · 2020-08-13T10:24:17Z

src/Symfony/Component/Scheduler/Messenger/TaskMessage.php

+     */
+    private $workerTimeout;
+
+    public function __construct(TaskInterface $task, $workerTimeout = 0.5)


watch out, sleep() waits an int, so need here int $workerTimeout = 1
or uses usleep() :)

fabpot · 2020-09-01T11:46:28Z

Closing this PR for now. @Guikingone and I will work together to submit a new PR with some substantial changes.

carsonbot added Status: Needs Review Feature labels Jan 12, 2020

Guikingone changed the title ~~[Feature] Scheduler~~ [Feature][Draft] Scheduler Jan 12, 2020

nicolas-grekas added this to the next milestone Jan 12, 2020

Guikingone requested a review from fabpot January 18, 2020 08:44

Guikingone removed the request for review from fabpot January 18, 2020 08:45

jderusse reviewed Jan 31, 2020

View reviewed changes

94noni reviewed Feb 12, 2020

View reviewed changes

Guikingone changed the title ~~[Feature][Draft] Scheduler~~ [Feature][Draft][WIP] Scheduler Feb 16, 2020

derrabus reviewed Aug 11, 2020

View reviewed changes

composer.json Outdated Show resolved Hide resolved

derrabus reviewed Aug 11, 2020

View reviewed changes

src/Symfony/Component/Scheduler/Bag/BagRegistry.php Outdated Show resolved Hide resolved

derrabus reviewed Aug 11, 2020

View reviewed changes

src/Symfony/Component/Scheduler/Bag/MailerBag.php Outdated Show resolved Hide resolved

Guikingone added 9 commits August 12, 2020 11:27

feat(Component): Scheduler

5bc76b7

refactor(bridge): progress on Kubernetes connection

ff617f2

feat(core): SchedulerAwareInterface added & configured

d675d5f

tests(core): tests improved for NullTaskRunner

4eeb6db

fix(bundle): UnusedTag fixed

c5452e8

refactor(bridge): Doctrine under work

4ec65ac

feat(bridge): Doctrine call under work

8772786

style(bridge): Doctrine connection fixed

bb83695

feat(bundle): FrameworkExtension improved with bridges services injec…

9336563

…tion

94noni reviewed Aug 12, 2020

View reviewed changes

Guikingone added 3 commits August 12, 2020 19:02

fix(bundle): fix on UnusedTagPass

e647da1

style(PR): fabbot fixs

1328d7d

style(PR): fabbot fixs

1321338

94noni reviewed Aug 12, 2020

View reviewed changes

Guikingone added 3 commits August 12, 2020 19:28

refactor(Scheduler): reviews

623b419

refactor(Scheduler): fixs

0686b03

refactor(Scheduler) reviews

34b3eba

94noni reviewed Aug 13, 2020

View reviewed changes

refactor(Scheduler): improvements|naming improved

e96a0ab

fabpot closed this Sep 1, 2020

nicolas-grekas modified the milestones: next, 5.2 Oct 5, 2020

Guikingone mentioned this pull request Jan 5, 2021

[Feature] Task scheduling #39719

Closed

xabbuh added the Scheduler label Mar 20, 2023

carsonbot changed the title ~~[Component] Scheduler~~ [Scheduler] [Component] Scheduler Mar 20, 2023


		public function onWorkerStarted(WorkerStartedEvent $event): void
		{

Uh oh!

[Scheduler] [Component] Scheduler #35315

[Scheduler] [Component] Scheduler #35315

Uh oh!

Conversation

Guikingone commented Jan 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Roadmap

Uh oh!

stephane-lou commented Jan 13, 2020

Uh oh!

Guikingone commented Jan 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kbond commented Jan 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Guikingone commented Jan 18, 2020

Uh oh!

fabpot commented Jan 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

monteiro commented Jan 19, 2020

Uh oh!

Guikingone commented Jan 19, 2020

Uh oh!

Guikingone commented Jan 23, 2020

Uh oh!

Guikingone commented Jan 23, 2020

Uh oh!

jderusse left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Guikingone commented Jan 31, 2020

Uh oh!

bigfoot90 commented Jan 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Guikingone commented Jan 31, 2020

Uh oh!

bigfoot90 commented Jan 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Guikingone commented Feb 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Guikingone commented Feb 3, 2020

Uh oh!

Guikingone commented Feb 3, 2020

Uh oh!

Guikingone commented Feb 4, 2020

Uh oh!

Guikingone commented Feb 6, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Guikingone commented Feb 16, 2020

Uh oh!

Guikingone commented Feb 17, 2020

Uh oh!

Guikingone commented Feb 18, 2020

Guikingone commented Jan 12, 2020 •

edited

Loading

Guikingone commented Jan 13, 2020 •

edited

Loading

kbond commented Jan 13, 2020 •

edited

Loading

fabpot commented Jan 18, 2020 •

edited

Loading

bigfoot90 commented Jan 31, 2020 •

edited

Loading

bigfoot90 commented Jan 31, 2020 •

edited

Loading

Guikingone commented Feb 2, 2020 •

edited

Loading