[HttpFoundation] added support for streamed responses #2935

fabpot · 2011-12-21T13:37:32Z

To stream a Response, use the StreamedResponse class instead of the
standard Response class:

$response = new StreamedResponse(function () {
    echo 'FOO';
});

$response = new StreamedResponse(function () {
    echo 'FOO';
}, 200, array('Content-Type' => 'text/plain'));

As you can see, a StreamedResponse instance takes a PHP callback instead of
a string for the Response content. It's up to the developer to stream the
response content from the callback with standard PHP functions like echo.
You can also use flush() if needed.

From a controller, do something like this:

$twig = $this->get('templating');

return new StreamedResponse(function () use ($templating) {
    $templating->stream('BlogBundle:Annot:streamed.html.twig');
}, 200, array('Content-Type' => 'text/html'));

If you are using the base controller, you can use the stream() method instead:

return $this->stream('BlogBundle:Annot:streamed.html.twig');

You can stream an existing file by using the PHP built-in readfile() function:

new StreamedResponse(function () use ($file) {
    readfile($file);
}, 200, array('Content-Type' => 'image/png');

Read http://php.net/flush for more information about output buffering in PHP.

Note that you should do your best to move all expensive operations to
be "activated/evaluated/called" during template evaluation.

Templates

If you are using Twig as a template engine, everything should work as
usual, even if are using template inheritance!

However, note that streaming is not supported for PHP templates. Support
is impossible by design (as the layout is rendered after the main content).

Exceptions

Exceptions thrown during rendering will be rendered as usual except that
some content might have been rendered already.

Limitations

As the getContent() method always returns false for streamed Responses, some
event listeners won't work at all:

Web debug toolbar is not available for such Responses (but the profiler works fine);
ESI is not supported.

Also note that streamed responses cannot benefit from HTTP caching for obvious
reasons.

To stream a Response, use the StreamedResponse class instead of the standard Response class: $response = new StreamedResponse(function () { echo 'FOO'; }); $response = new StreamedResponse(function () { echo 'FOO'; }, 200, array('Content-Type' => 'text/plain')); As you can see, a StreamedResponse instance takes a PHP callback instead of a string for the Response content. It's up to the developer to stream the response content from the callback with standard PHP functions like echo. You can also use flush() if needed. From a controller, do something like this: $twig = $this->get('templating'); return new StreamedResponse(function () use ($templating) { $templating->stream('BlogBundle:Annot:streamed.html.twig'); }, 200, array('Content-Type' => 'text/html')); If you are using the base controller, you can use the stream() method instead: return $this->stream('BlogBundle:Annot:streamed.html.twig'); You can stream an existing file by using the PHP built-in readfile() function: new StreamedResponse(function () use ($file) { readfile($file); }, 200, array('Content-Type' => 'image/png'); Read http://php.net/flush for more information about output buffering in PHP. Note that you should do your best to move all expensive operations to be "activated/evaluated/called" during template evaluation. Templates --------- If you are using Twig as a template engine, everything should work as usual, even if are using template inheritance! However, note that streaming is not supported for PHP templates. Support is impossible by design (as the layout is rendered after the main content). Exceptions ---------- Exceptions thrown during rendering will be rendered as usual except that some content might have been rendered already. Limitations ----------- As the getContent() method always returns false for streamed Responses, some event listeners won't work at all: * Web debug toolbar is not available for such Responses (but the profiler works fine); * ESI is not supported. Also note that streamed responses cannot benefit from HTTP caching for obvious reasons.

lsmith77 · 2011-12-21T13:47:08Z

src/Symfony/Bundle/FrameworkBundle/HttpKernel.php

@@ -146,7 +147,11 @@ public function render($controller, array $options = array())
                throw new \RuntimeException(sprintf('Error when rendering "%s" (Status code is %s).', $request->getUri(), $response->getStatusCode()));
            }

-            return $response->getContent();
+            if ($response instanceof StreamedResponse) {


imho it would be nicer to check the negation to return early and to get rid of the else

Seldaek · 2011-12-21T14:34:13Z

Just an idea: what about exposing flush() to twig? Possibly in a way that it will not call it if the template is not streaming. That way you could always add a flush() after your tag to make sure that goes out as fast as possible, but it wouldn't mess with non-streamed responses. Although it appears flush() doesn't affect output buffers, so I guess it doesn't need anything special.

When you say "ESI is not supported.", that means only the AppCache right? I don't see why this would affect Varnish, but then again as far as I know Varnish will buffer if ESI is used so the benefit of streaming there is non-existent.

cordoval · 2011-12-21T16:04:21Z

wonder what the use case is for streaming a response, very interesting.

johnkary · 2011-12-21T16:19:48Z

@cordoval Common use cases are present fairly well by this RailsCast video: http://railscasts.com/episodes/266-http-streaming

Essentially it allows faster fetching of web assets (JS, CSS, etc) located in the <head></head>, allowing those assets to be fetched as soon as possible before the remainder of the content body is computed and sent to the browser. The end goal is to improve page load speed.

There are other uses cases too like making large body content available quickly to the service consuming it. Think if you were monitoring a live feed of JSON data of newest Twitter comments.

lsmith77 · 2011-12-21T16:54:35Z

How does this relate the limitations mentioned in:
http://yehudakatz.com/2010/09/07/automatic-flushing-the-rails-3-1-plan/

Am I right to understand that due to how twig works we are not really streaming the content pieces when we call render(), but instead the entire template with its layout is rendered and only then will we flush? or does it mean that the render call will work its way to the top level layout template and form then on it can send the content until it hits another block, which it then first renders before it continues to send the data?

stof · 2011-12-21T17:02:53Z

@lsmith77 this is why the stream method calls display in Twig instead of render. display uses echo to print the output of the template line by line (and blocks are simply method calls in the middle). Look at your compiled templates to see it (the doDisplay method)
Rendering a template with Twig simply use an output buffer around the rendering.

fabpot · 2011-12-21T17:24:33Z

@lsmith77: We don't have the Rails problem thanks to Twig as the order of execution is the right one by default (the layout is executed first); it means that we can have the flush feature without any change to how the core works. As @stof mentioned, we are using display, not render, so we are streaming your templates for byte one.

fabpot · 2011-12-21T17:36:41Z

@Seldaek: yes, I meant ESI with the PHP reverse proxy.

fabpot · 2011-12-21T17:37:34Z

@Seldaek: I have flush() support for Twig on my todo-list. As you mentioned, It should be trivial to implement.

vicb · 2011-12-21T17:39:14Z

src/Symfony/Component/HttpFoundation/StreamedResponse.php

+
+        $this->streamed = true;
+
+        if (!is_callable($this->callback)) {


should this test be moved to the ctor ?

fzaninotto · 2011-12-21T17:48:18Z

How do streaming responses deal with assets that must be called in the head, but are declared in the body?

fabpot · 2011-12-21T17:52:12Z

@fzaninotto: What do you mean?

With Twig, your layout is defined with blocks ("holes"). These blocks are overridden by child templates, but evaluated as they are encountered in the layout. So, everything works as expected.

As noted in the commit message, this does not work with PHP templates for the problems mentioned in the Rails post (as the order of execution is not the right one -- the child template is first evaluated and then the layout).

fzaninotto · 2011-12-21T18:07:35Z

I was referring to using Assetic. Not sure if this compiles to Twig the same way as javascript and stylesheet blocks placed in the head - and therefore executed in the right way.

fabpot · 2011-12-21T18:34:59Z

@Seldaek: I've just added a flush tag in Twig 1.5: fabpot/Twig@1d6dfad

catchamonkey · 2011-12-21T21:29:22Z

I'm really happy you've got this into the core, it's a great feature to have! Good work.

vicb · 2011-12-21T22:00:55Z

src/Symfony/Component/HttpFoundation/StreamedResponse.php

+            $this->headers->set('Transfer-Encoding', 'chunked');
+        }
+
+        $this->headers->set('Cache-Control', 'no-cache');


There is probably a good reason to avoid caching but I can not figure out, any hint ?

semantic: maybe this should be moved after the call to the parent method ?

… creation

…keep BC with 2.0

@cordoval

Commits ------- 887c0e9 moved EngineInterface::stream() to a new StreamingEngineInterface to keep BC with 2.0 473741b added the possibility to change a StreamedResponse callback after its creation 8717d44 moved a test in the constructor e44b8ba made some cosmetic changes 0038d1b [HttpFoundation] added support for streamed responses Discussion ---------- [HttpFoundation] added support for streamed responses To stream a Response, use the StreamedResponse class instead of the standard Response class: $response = new StreamedResponse(function () { echo 'FOO'; }); $response = new StreamedResponse(function () { echo 'FOO'; }, 200, array('Content-Type' => 'text/plain')); As you can see, a StreamedResponse instance takes a PHP callback instead of a string for the Response content. It's up to the developer to stream the response content from the callback with standard PHP functions like echo. You can also use flush() if needed. From a controller, do something like this: $twig = $this->get('templating'); return new StreamedResponse(function () use ($templating) { $templating->stream('BlogBundle:Annot:streamed.html.twig'); }, 200, array('Content-Type' => 'text/html')); If you are using the base controller, you can use the stream() method instead: return $this->stream('BlogBundle:Annot:streamed.html.twig'); You can stream an existing file by using the PHP built-in readfile() function: new StreamedResponse(function () use ($file) { readfile($file); }, 200, array('Content-Type' => 'image/png'); Read http://php.net/flush for more information about output buffering in PHP. Note that you should do your best to move all expensive operations to be "activated/evaluated/called" during template evaluation. Templates --------- If you are using Twig as a template engine, everything should work as usual, even if are using template inheritance! However, note that streaming is not supported for PHP templates. Support is impossible by design (as the layout is rendered after the main content). Exceptions ---------- Exceptions thrown during rendering will be rendered as usual except that some content might have been rendered already. Limitations ----------- As the getContent() method always returns false for streamed Responses, some event listeners won't work at all: * Web debug toolbar is not available for such Responses (but the profiler works fine); * ESI is not supported. Also note that streamed responses cannot benefit from HTTP caching for obvious reasons. --------------------------------------------------------------------------- by Seldaek at 2011/12/21 06:34:13 -0800 Just an idea: what about exposing flush() to twig? Possibly in a way that it will not call it if the template is not streaming. That way you could always add a flush() after your </head> tag to make sure that goes out as fast as possible, but it wouldn't mess with non-streamed responses. Although it appears flush() doesn't affect output buffers, so I guess it doesn't need anything special. When you say "ESI is not supported.", that means only the AppCache right? I don't see why this would affect Varnish, but then again as far as I know Varnish will buffer if ESI is used so the benefit of streaming there is non-existent. --------------------------------------------------------------------------- by cordoval at 2011/12/21 08:04:21 -0800 wonder what the use case is for streaming a response, very interesting. --------------------------------------------------------------------------- by johnkary at 2011/12/21 08:19:48 -0800 @cordoval Common use cases are present fairly well by this RailsCast video: http://railscasts.com/episodes/266-http-streaming Essentially it allows faster fetching of web assets (JS, CSS, etc) located in the <head></head>, allowing those assets to be fetched as soon as possible before the remainder of the content body is computed and sent to the browser. The end goal is to improve page load speed. There are other uses cases too like making large body content available quickly to the service consuming it. Think if you were monitoring a live feed of JSON data of newest Twitter comments. --------------------------------------------------------------------------- by lsmith77 at 2011/12/21 08:54:35 -0800 How does this relate the limitations mentioned in: http://yehudakatz.com/2010/09/07/automatic-flushing-the-rails-3-1-plan/ Am I right to understand that due to how twig works we are not really streaming the content pieces when we call render(), but instead the entire template with its layout is rendered and only then will we flush? or does it mean that the render call will work its way to the top level layout template and form then on it can send the content until it hits another block, which it then first renders before it continues to send the data? --------------------------------------------------------------------------- by stof at 2011/12/21 09:02:53 -0800 @lsmith77 this is why the ``stream`` method calls ``display`` in Twig instead of ``render``. ``display`` uses echo to print the output of the template line by line (and blocks are simply method calls in the middle). Look at your compiled templates to see it (the ``doDisplay`` method) Rendering a template with Twig simply use an output buffer around the rendering. --------------------------------------------------------------------------- by fabpot at 2011/12/21 09:24:33 -0800 @lsmith77: We don't have the Rails problem thanks to Twig as the order of execution is the right one by default (the layout is executed first); it means that we can have the flush feature without any change to how the core works. As @stof mentioned, we are using `display`, not `render`, so we are streaming your templates for byte one. --------------------------------------------------------------------------- by fabpot at 2011/12/21 09:36:41 -0800 @Seldaek: yes, I meant ESI with the PHP reverse proxy. --------------------------------------------------------------------------- by fabpot at 2011/12/21 09:37:34 -0800 @Seldaek: I have `flush()` support for Twig on my todo-list. As you mentioned, It should be trivial to implement. --------------------------------------------------------------------------- by fzaninotto at 2011/12/21 09:48:18 -0800 How do streaming responses deal with assets that must be called in the head, but are declared in the body? --------------------------------------------------------------------------- by fabpot at 2011/12/21 09:52:12 -0800 @fzaninotto: What do you mean? With Twig, your layout is defined with blocks ("holes"). These blocks are overridden by child templates, but evaluated as they are encountered in the layout. So, everything works as expected. As noted in the commit message, this does not work with PHP templates for the problems mentioned in the Rails post (as the order of execution is not the right one -- the child template is first evaluated and then the layout). --------------------------------------------------------------------------- by fzaninotto at 2011/12/21 10:07:35 -0800 I was referring to using Assetic. Not sure if this compiles to Twig the same way as javascript and stylesheet blocks placed in the head - and therefore executed in the right way. --------------------------------------------------------------------------- by fabpot at 2011/12/21 10:34:59 -0800 @Seldaek: I've just added a `flush` tag in Twig 1.5: fabpot/Twig@1d6dfad --------------------------------------------------------------------------- by catchamonkey at 2011/12/21 13:29:22 -0800 I'm really happy you've got this into the core, it's a great feature to have! Good work.

b00gizm · 2011-12-31T11:44:00Z

src/Symfony/Component/Templating/PhpEngine.php

@@ -28,7 +28,7 @@
 *
 * @api
 */
-class PhpEngine implements EngineInterface, \ArrayAccess
+class PhpEngine implements EngineInterface, StreamingEngineInterface, \ArrayAccess


Is there a reason why the PhpEngine implements the StreamingEngineInterface just to throw an Exception? Didn't you say that streaming responses are impossible for PHP templates by design?

You are right. This is a nice side-effect of having another interface for streaming. Fixed here: e462f7b

igorw · 2011-12-31T19:45:15Z

I ran into some problems with the way this is implemented. The issues are related to transfer-encoding.

First, it is not possible to force identity (the default) encoding, because StreamedResponse overrides it in any case. It should only set the encoding if none is set yet. Note: Even if it is explicitly set to identity and not overriden to chunked, apache will change it to "identity, chunked" because no content-length is set.

Now this brings us to the second issue. When you send a transfer-encoding=chunked header, apache will expect the content to already be chunked. Of course this is not the case, and thus it does not work. Unless other webservers require it to be present, I would suggest simply not setting the transfer-encoding header. Since no content-length is set, the server should switch to chunked automatically.

igorw · 2011-12-31T19:57:13Z

Nginx does not have that problem. But: Proposed solution confirmed working with nginx 1.0.5.

Commits ------- 7ae9348 [streaming] Document and test that Transfer-Encoding is absent 83c23ca [streaming] Do not set a Transfer-Encoding header of chunked Discussion ---------- [HttpFoundation] Do not set a Transfer-Encoding header of chunked for streaming responses Bug fix: yes Feature addition: no Backwards compatibility break: no Symfony2 tests pass: yes This is an adjustment to the streaming introduced in #2935. Apache expects the response to already be in chunked format when the Transfer-Encoding header is set to chunked, which causes it to not deliver the streamed body. If no Content-Length is set on the response (regardless of Transfer-Encoding), web servers will automatically switch to chunked Transfer-Encoding, and handle the chunking for you. I have tested this with Apache 2.2.2 and nginx 1.0.5. Testing with other webservers is appreciated.

Commits ------- 8dd779e [stream] Fix rst formatting 73fe0b1 [stream] Add a nice API for streaming responses Discussion ---------- [stream] Add a nice API for streaming responses This PR adds a nice API for creating streaming responses with the new StreamedResponse class that was added to symfony [just recently](symfony/symfony#2935). Usage is documented in the usage doc, but for lazy people: ```php <?php $app->get('/images/{file}', function ($file) use ($app) { if (!file_exists(__DIR__.'/images/'.$file)) { return $app->abort(404, 'The image was not found.'); } $stream = function () use ($file) { readfile($file); }; return $app->stream($stream, 200, array('Content-Type' => 'image/png')); }); ``` Note: I had to point the autoloader to a copy of symfony/symfony because the subtree splits are not up-to-date. Thus the vendors are out of date too right now.

This PR was squashed before being merged into the 2.7 branch (closes #24626). Discussion ---------- streamed response should return $this | Q | A | ------------- | --- | Branch? | 2.7 | Bug fix? | yes | New feature? | no | BC breaks? | may be yes? | Deprecations? | no | Tests pass? | yes | License | MIT --- `sendHeaders()` and `sendContent()` should return $this, as in the parent class. related PRs: #2935 #20289 Commits ------- 058fb84 streamed response should return $this

@alexander-schranz

…rves no purpose anymore (nicolas-grekas) This PR was merged into the 6.1 branch. Discussion ---------- [HttpKernel] Deprecate StreamedResponseListener, it serves no purpose anymore | Q | A | ------------- | --- | Branch? | 6.1 | Bug fix? | no | New feature? | no | Deprecations? | yes | Tickets | - | License | MIT | Doc PR | - `StreamedResponseListener` has been introduced at the same time as `StreamedResponse` in #2935. Its purpose was to make catching exceptions easier by wrapping the call to `$response->send()` in the main try/catch of `HttpKernel`. Since #12998, we have `HttpKernel::terminateWithException()`, and we don't need that anymore, so we can just remove the listener. This will help [integrate Symfony into e.g. Swoole](php-runtime/runtime#115) /cc @alexander-schranz. Commits ------- ee61774 [HttpKernel] Deprecate StreamedResponseListener, it serves no purpose anymore

lsmith77 reviewed Dec 21, 2011
View reviewed changes

made some cosmetic changes

e44b8ba

vicb reviewed Dec 21, 2011
View reviewed changes

moved a test in the constructor

8717d44

vicb reviewed Dec 21, 2011
View reviewed changes

fabpot added 2 commits December 22, 2011 07:58

added the possibility to change a StreamedResponse callback after its…

473741b

… creation

moved EngineInterface::stream() to a new StreamingEngineInterface to …

887c0e9

…keep BC with 2.0

fabpot merged commit 887c0e9 into master Dec 31, 2011

b00gizm reviewed Dec 31, 2011
View reviewed changes

igorw mentioned this pull request Jan 2, 2012

[HttpFoundation] Do not set a Transfer-Encoding header of chunked for streaming responses #3017

Merged

igorw mentioned this pull request Jan 24, 2012

[2.1] Transport agnostic HTTP streaming #3177

Closed

ogizanagi mentioned this pull request May 29, 2015

[HttpKernel] Why are StreamedResponse's sent by a kernel.response listener? #13591

Closed

jakzal mentioned this pull request Feb 8, 2017

StreamedResponses send response to stdout before $response->send() is called in app.php #21355

Closed

DQNEO mentioned this pull request Oct 19, 2017

streamed response should return $this #24626

Closed

nicolas-grekas mentioned this pull request Feb 18, 2022

[HttpKernel] Deprecate StreamedResponseListener, it serves no purpose anymore #45476

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HttpFoundation] added support for streamed responses #2935

[HttpFoundation] added support for streamed responses #2935

fabpot commented Dec 21, 2011

lsmith77 Dec 21, 2011

fabpot Dec 21, 2011

Seldaek commented Dec 21, 2011

cordoval commented Dec 21, 2011

johnkary commented Dec 21, 2011

lsmith77 commented Dec 21, 2011

stof commented Dec 21, 2011

fabpot commented Dec 21, 2011

fabpot commented Dec 21, 2011

fabpot commented Dec 21, 2011

vicb Dec 21, 2011

fabpot Dec 21, 2011

fzaninotto commented Dec 21, 2011

fabpot commented Dec 21, 2011

fzaninotto commented Dec 21, 2011

fabpot commented Dec 21, 2011

catchamonkey commented Dec 21, 2011

vicb Dec 21, 2011

b00gizm Dec 31, 2011

fabpot Dec 31, 2011

igorw commented Dec 31, 2011

igorw commented Dec 31, 2011

[HttpFoundation] added support for streamed responses #2935

[HttpFoundation] added support for streamed responses #2935

Conversation

fabpot commented Dec 21, 2011

Templates

Exceptions

Limitations

lsmith77 Dec 21, 2011

Choose a reason for hiding this comment

fabpot Dec 21, 2011

Choose a reason for hiding this comment

Seldaek commented Dec 21, 2011

cordoval commented Dec 21, 2011

johnkary commented Dec 21, 2011

lsmith77 commented Dec 21, 2011

stof commented Dec 21, 2011

fabpot commented Dec 21, 2011

fabpot commented Dec 21, 2011

fabpot commented Dec 21, 2011

vicb Dec 21, 2011

Choose a reason for hiding this comment

fabpot Dec 21, 2011

Choose a reason for hiding this comment

fzaninotto commented Dec 21, 2011

fabpot commented Dec 21, 2011

fzaninotto commented Dec 21, 2011

fabpot commented Dec 21, 2011

catchamonkey commented Dec 21, 2011

vicb Dec 21, 2011

Choose a reason for hiding this comment

b00gizm Dec 31, 2011

Choose a reason for hiding this comment

fabpot Dec 31, 2011

Choose a reason for hiding this comment

igorw commented Dec 31, 2011

igorw commented Dec 31, 2011