[HttpClient] add EventSourceHttpClient to consume Server-Sent Events #36692

soyuka · 2020-05-04T17:17:55Z

Q	A
Branch?	master
Bug fix?	no
New feature?	yes
Deprecations?	no
Tickets	na
License	MIT
Doc PR	symfony/symfony-docs#...

First implementation

This patch implements the [w3c Server-Sent Events specification](https://www.w3.org/TR/eventsource/#eventsource) on top of symfony's http client. It provides an `EventSource` class that allows you to interact of server sent events.

Comparing to the Javascript implementation, we won't be able to use the same API. Indeed, in php listeners need to be setup before we connect to the HTTP stream.
I'm not fond of adding a dependency to EventDispatcher from HTTP Client, therefore I'm all ears if you have better solutions.
About event parsing, I wanted to avoid using regular expression and it uses smart data split. Note that I had to concatenate an internal buffer and only handle the data when a newline is found to cover long chunks. This is an alternative to this react php eventsource. Note that this implementation is closer to the specification in some cases that are still to be covered by tests (retry, data:value without space after colon is valid etc.).

This is an implementation of the Server-Sent Events specification based on symfony's HTTP Client. After a few suggestions on the first implementation (see details above), I've implemented a chunk generator with this kind of API:

$client = new EventSourceHttpClient($client, 10);
$source = $client->connect('GET', "http://localhost:8080/events");
while($source) {
    foreach ($client->stream($source, 2) as $r => $chunk) {
        if ($chunk->isTimeout()) {
            dump([
                'timeout' => [
                    'retry' => 1 + count($r->getInfo('previous_info') ?? [])
                ],
            ]);
            continue;
        }
        if ($chunk->isLast()) {
            dump([
                'eof' => [
                    'retries' => count($r->getInfo('previous_info') ?? [])
                ],
            ]);
            $source = null;
            return;
        }

        dump($chunk);
    }
}

TODO:

validate implementation (~~don't use EventDispatcher ?~~, need to be implemented as stream instead of message)
default timeout value
implement retry/reconnection
tests (do test with super long chunk, retry, bad http content-type response)
update changelog
document

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php

src/Symfony/Component/HttpClient/EventSource.php

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php

stof · 2020-05-04T18:54:35Z

Is ->connect() blocking until the event source is over ? Or how does this work to keep running while checking for data ?

Both JS and ReactPHP have an event loop running, and the checks in the EventSource are happening in this event loop. But without an event loop, I'm not sure this will be as easy to use.

soyuka · 2020-05-04T20:44:44Z

Is ->connect() blocking until the event source is over ?

For now it is yes. It just opens an http connection open and uses a stream to get the data. On timeout, it reconnects to that http connection.
An event loop would definitely be appropriate. Even without I see use cases though.

nicolas-grekas · 2020-05-04T20:49:07Z

I would suggest a radically different approach, based on the same design as HttpClient: a chunk generator.

src/Symfony/Component/HttpClient/EventSource.php

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php

dunglas · 2020-05-04T21:46:16Z

You should implement exactly the algorithm described in the WHATWG spec (9.2.4 and 9.2.5): https://html.spec.whatwg.org/multipage/server-sent-events.html#parsing-an-event-stream

dunglas · 2020-05-04T22:35:55Z

I would suggest a radically different approach, based on the same design as HttpClient: a chunk generator.

I agree. It could be a class decorating HttpClient and will an API like this:

$response = $client->request('GET', 'https://demo.mercure.rocks/.well-known/mercure?topic=https://example.com');

if (200 !== $response->getStatusCode()) {
    throw new \Exception('...');
}

// messages implement Symfony\Contracts\HttpClient\EventInterface
$fileHandler = fopen('/ubuntu.iso', 'w');
foreach ($client->messages($response) as $message) {
    // https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#Event_stream_format
    // https://github.com/symfony/mercure/blob/master/src/Update.php#L26
    $message->getData(); // string
    $message->getType(); // ?string
    $message->getId(); // ?string
    $message->getRetry(); // ?string

    // When you're done, close the connection with the server
    $response->cancel();
}

src/Symfony/Component/HttpClient/Chunk/MessageChunk.php

src/Symfony/Component/HttpClient/EventSourceHttpClient.php

src/Symfony/Component/HttpClient/Tests/Chunk/MessageChunkTest.php

dunglas · 2020-05-05T09:05:14Z

src/Symfony/Component/HttpClient/Tests/EventSourceHttpClientTest.php

+
+$es = new EventSourceHttpClient(HttpClient::create());
+
+$response = $es->request('GET', 'http://localhost:8080/events');


We already have Vulcain in the CI to test HttpClient, you could add Mercure, it uses all the spec capabilities.

src/Symfony/Component/HttpClient/Chunk/MessageChunk.php

nicolas-grekas

Don't we want to be able to read a stream in a non-blocking way?
HttpClientInterface::stream() allows doing so by passing $timeout = 0.

src/Symfony/Component/HttpClient/ServerSentEvents/EventSource.php

nicolas-grekas

Really nice :)
Just minor comments on my side.

src/Symfony/Component/HttpClient/Chunk/ServerSentEvent.php

src/Symfony/Component/HttpClient/EventSourceHttpClient.php

nicolas-grekas · 2020-06-12T11:07:33Z

src/Symfony/Component/HttpClient/EventSourceHttpClient.php

+        $this->reconnectionTime = $reconnectionTime;
+    }
+
+    public function connect($method, string $url, array $options = []): ResponseInterface


Would it make sense to have an interface for this one?
I'm not sure, just wondering.

I don't think so but let's gather opinions.

I'm also in favor of having a dedicated interface. The user may want to provide its own EventSource implementation (for instance, to support a server not respecting strictly the standard or something like that).

I think I'd prefer to have the new interface in the component, not in the contracts.

src/Symfony/Component/HttpClient/EventSourceHttpClient.php

src/Symfony/Component/HttpClient/Tests/EventSourceHttpClientTest.php

nicolas-grekas · 2020-06-12T11:31:19Z

What about adding an autowiring alias btw?

soyuka · 2020-06-12T11:54:46Z

What about adding an autowiring alias btw?

Now I think having an interface would make sense :p. If we do so should we also add a configuration for the reconnectionTime? What about doing so in another PR?

nicolas-grekas · 2020-06-12T11:58:53Z

Another PR works. We need to gain some experience to know what's the best way to consume this class.

src/Symfony/Component/HttpClient/EventSourceHttpClient.php

src/Symfony/Component/HttpClient/Response/AsyncResponse.php

stof · 2020-06-12T12:46:04Z

src/Symfony/Component/HttpClient/EventSourceHttpClient.php

+            }
+
+            if ($chunk->isLast()) {
+                yield $chunk;


is it possible for the last chunk to have a non-empty content ? If yes, shouldn't we recreate a new last chunk instead, excluding the content we processed for new chunks ?

also, is it possible to reach the last chunk with some remaining buffered content in the state ? and what happens in that case ?

I'd say that it's not really a problem and if the last chunk is partial content it won't be transformed as a ServerSentEvent chunk (which parses only full chunks). The full data will be retrieved later by using a Last-Event-Id and there's no need to process it here.

The last chunk is always empty so that's OK. About the remaining data in the buffer, L122 above ensures it's always emptied when the last chunk arrives.

stof · 2020-06-12T13:03:11Z

src/Symfony/Component/HttpClient/EventSourceHttpClient.php

+        ], true));
+    }
+
+    public function request(string $method, string $url, array $options = []): ResponseInterface


I'm wondering whether it make sense for an EventSource implementation to still be a full HttpClient as well. Shouldn't it use a client inside it (a special one) instead ?

Also, will code using ->connect() receive other chunks than event ones ?

Also, will code using ->connect() receive other chunks than event ones ?

Yes it does, full power to the end-user. As the API exposed here is rather low-level, having a full HttpClient makes things easier. Also, wrapping a special client inside a new class would add a rather useless class to this component in my opinion.

Also, having this implement HttpClientInterface allows streaming both regular requests and event-streams at the same time.

src/Symfony/Component/HttpClient/EventSourceHttpClient.php

fabpot · 2020-08-06T06:58:59Z

@soyuka Can you work on a PR for the docs?

fabpot · 2020-08-06T06:59:22Z

Thank you @soyuka.

This PR was merged into the master branch. Discussion ---------- [HttpClient] Document EventSourceHttpClient Documenting symfony/symfony#36692 Commits ------- 1bfa68f Document EventSourceHttpClient

carsonbot added Status: Needs Review Feature labels May 4, 2020

soyuka force-pushed the feat/eventsource branch from e5b152b to c6983fe Compare May 4, 2020 17:17

Pierstoval reviewed May 4, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php Outdated Show resolved Hide resolved

Pierstoval reviewed May 4, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php Outdated Show resolved Hide resolved

stof reviewed May 4, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSource.php Outdated Show resolved Hide resolved

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php Outdated Show resolved Hide resolved

nicolas-grekas added this to the next milestone May 4, 2020

michaljusiega reviewed May 4, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSource.php Outdated Show resolved Hide resolved

dunglas reviewed May 4, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSource.php Outdated Show resolved Hide resolved

dunglas reviewed May 4, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php Outdated Show resolved Hide resolved

dunglas reviewed May 4, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php Outdated Show resolved Hide resolved

dunglas reviewed May 4, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSource/MessageEvent.php Outdated Show resolved Hide resolved

dunglas closed this May 4, 2020

dunglas reopened this May 4, 2020

soyuka force-pushed the feat/eventsource branch 2 times, most recently from de989de to b88c4ca Compare May 5, 2020 08:08

dunglas requested changes May 5, 2020

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Needs Review labels May 5, 2020

dunglas reviewed May 5, 2020

View reviewed changes

src/Symfony/Component/HttpClient/Chunk/MessageChunk.php Outdated Show resolved Hide resolved

soyuka force-pushed the feat/eventsource branch 3 times, most recently from addda43 to 9bb244e Compare May 5, 2020 11:19

nicolas-grekas reviewed May 5, 2020

View reviewed changes

soyuka force-pushed the feat/eventsource branch from 746abe2 to c539207 Compare May 7, 2020 07:31

nicolas-grekas reviewed Jun 12, 2020

View reviewed changes

stof reviewed Jun 12, 2020

View reviewed changes

soyuka force-pushed the feat/eventsource branch from 522a4b7 to e923244 Compare June 13, 2020 16:08

nicolas-grekas approved these changes Jun 23, 2020

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Work labels Jun 23, 2020

nicolas-grekas force-pushed the feat/eventsource branch from e923244 to 9c1f354 Compare July 4, 2020 15:19

nicolas-grekas mentioned this pull request Jul 7, 2020

[HttpClient] Make the client aware of throttling / rate limits #37471

Closed

dunglas reviewed Jul 15, 2020

View reviewed changes

src/Symfony/Component/HttpClient/EventSourceHttpClient.php Outdated Show resolved Hide resolved

dunglas reviewed Jul 15, 2020

View reviewed changes

soyuka force-pushed the feat/eventsource branch 3 times, most recently from cb05755 to 25387a5 Compare July 16, 2020 08:21

dunglas approved these changes Jul 16, 2020

View reviewed changes

maxhelias approved these changes Aug 5, 2020

View reviewed changes

chalasr approved these changes Aug 5, 2020

View reviewed changes

[HttpClient] add EventSourceHttpClient to consume Server-Sent Events

12ccca3

fabpot force-pushed the feat/eventsource branch from 25387a5 to 12ccca3 Compare August 6, 2020 06:59

fabpot mentioned this pull request Aug 6, 2020

[HttpClient] add EventSourceHttpClient to consume Server-Sent Events symfony/symfony-docs#14031

Closed

fabpot merged commit d66a0a7 into symfony:master Aug 6, 2020

soyuka mentioned this pull request Aug 6, 2020

[HttpClient] Document EventSourceHttpClient symfony/symfony-docs#14032

Merged

nicolas-grekas modified the milestones: next, 5.2 Oct 5, 2020

fabpot mentioned this pull request Oct 5, 2020

Release v5.2.0-BETA1 #38417

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HttpClient] add EventSourceHttpClient to consume Server-Sent Events #36692

[HttpClient] add EventSourceHttpClient to consume Server-Sent Events #36692

soyuka commented May 4, 2020 •

edited

Loading

stof commented May 4, 2020

soyuka commented May 4, 2020

nicolas-grekas commented May 4, 2020 •

edited

Loading

dunglas commented May 4, 2020

dunglas commented May 4, 2020 •

edited

Loading

dunglas May 5, 2020

nicolas-grekas left a comment

nicolas-grekas left a comment

nicolas-grekas Jun 12, 2020

soyuka Jun 12, 2020

dunglas Jul 15, 2020

nicolas-grekas Jul 15, 2020 •

edited

Loading

nicolas-grekas commented Jun 12, 2020

soyuka commented Jun 12, 2020

nicolas-grekas commented Jun 12, 2020

stof Jun 12, 2020

stof Jun 12, 2020

soyuka Jun 12, 2020

nicolas-grekas Jun 12, 2020

stof Jun 12, 2020

soyuka Jun 12, 2020

nicolas-grekas Jun 12, 2020 •

edited

Loading

fabpot commented Aug 6, 2020

fabpot commented Aug 6, 2020


		$es = new EventSourceHttpClient(HttpClient::create());

		$response = $es->request('GET', 'http://localhost:8080/events');

[HttpClient] add EventSourceHttpClient to consume Server-Sent Events #36692

[HttpClient] add EventSourceHttpClient to consume Server-Sent Events #36692

Conversation

soyuka commented May 4, 2020 • edited Loading

stof commented May 4, 2020

soyuka commented May 4, 2020

nicolas-grekas commented May 4, 2020 • edited Loading

dunglas commented May 4, 2020

dunglas commented May 4, 2020 • edited Loading

Choose a reason for hiding this comment

nicolas-grekas left a comment

Choose a reason for hiding this comment

nicolas-grekas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolas-grekas Jul 15, 2020 • edited Loading

Choose a reason for hiding this comment

nicolas-grekas commented Jun 12, 2020

soyuka commented Jun 12, 2020

nicolas-grekas commented Jun 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolas-grekas Jun 12, 2020 • edited Loading

Choose a reason for hiding this comment

fabpot commented Aug 6, 2020

fabpot commented Aug 6, 2020

soyuka commented May 4, 2020 •

edited

Loading

nicolas-grekas commented May 4, 2020 •

edited

Loading

dunglas commented May 4, 2020 •

edited

Loading

nicolas-grekas Jul 15, 2020 •

edited

Loading

nicolas-grekas Jun 12, 2020 •

edited

Loading