[RFC][HttpKernel] Make downstream HTTP caches expire on deployments #31007

mpdude · 2019-04-08T12:35:09Z

In several projects, we are using Content Delivery Networks in front of Symfony applications. In those cases, most of the controllers are able to send Last-Modified headers along with the responses. Often this is derived from some database information, for example the time a record has last been saved. Also, there is some infrastructure (bundles, annotations) in place to make validation via If-Modified-Since easy.

But there is one case where we've struggled a few times already: When we deploy a new version of the application, there's nothing that makes sure that "old" responses are no longer served from downstream caches.

I am not talking about a "grace" period where the s-maxage runs down, but the fact that the downstream cache can successfully re-validate the stale cache entry since my application is not aware of the new deployment (it just looks at the database records). Of course, I do not want to have to deal with deployment timestamps in the controllers either.

One solution would be to force a cache purge. But that's an annoying task and can be quite slow in CDNs.

So, one idea I came up with was to add something along the following lines in our AppKernel:

use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpKernel\HttpKernelInterface;

class AppKernel ...
{
    public function handle(Request $request, $type = HttpKernelInterface::MASTER_REQUEST, $catch = true)
    {
        $this->boot();

        if (!$this->getContainer()->hasParameter('project.deployment_ts')) {
            return parent::handle($request, $type, $catch);
        }

        $lastDeploymentTimestamp = $this->getContainer()->getParameter('project.deployment_ts');

        $ifModifiedSince = $request->headers->get('if-modified-since');
        if ($ifModifiedSince !== null && strtotime($ifModifiedSince) < $lastDeploymentTimestamp) {
            $request->headers->remove('if-modified-since');
        }

        $response = parent::handle($request, $type, $catch);

        $lastmod = $response->getLastModified();
        if ($lastmod !== null) {
            $lastDeployment = new \DateTime('@'.$lastDeploymentTimestamp);

            if ($lastDeployment > $lastmod) {
                $response->setLastModified($lastDeployment);
            }
        }

        return $response;
    }
}

This checks if there is a project.deployment_ts parameter in the DIC which is a unix timestamp. Somehow your deployment tooling has to set or provide this parameter, but that's probably not an issue.

The code will then make sure that If-Modified-Since conditional requests will return complete 200 responses if the given time is before the current deployment. Also, it will update the Last-Modified header to at least this time.

I'd love to get feedback on the issue. Have you encountered a similar problem as well? If there is enough support and interest, I would see how this could be fit into the Kernel or the HttpKernel component.

Update

Just realized that this has nothing to do with the CDN. It affects any validation-based downstream cache, including browser caches that cannot be PURGEd at all.

The text was updated successfully, but these errors were encountered:

nicolas-grekas · 2019-04-08T13:09:10Z

There are three parameters that could help achieve this:
'container.build_hash'
'container.build_id'
'container.build_time'

Would that be enough?

mpdude · 2019-04-08T13:18:26Z

Great, wasn’t aware of them. So that would be solved out of the box.

The remaining question then is: Should the HttpKernel take control of Last-Modified information in this way, possibly overriding the value set in controllers?

What about ETag-based validation?

stof · 2019-04-08T13:28:55Z

Expiring the ETag-based validation would involve modifying the Etag generated by the controller, to include some info related to the build in it, alongside the original value.

But if your Etag is based on a hash of the content, this might not be needed.

Note that using an event listener might be easier to maintain than doing that directly in the kernel

javiereguiluz · 2019-04-08T14:37:45Z

@mpdude those parameters weren't documented, so we're adding some docs form them in symfony/symfony-docs#11361

mpdude · 2019-04-08T15:17:41Z

Now that I see what they were added for, I am not so sure anymore we can use them for this purpose:

Imagine events happen in this order:

New (Docker) image of the application is built, including the cache/container and compiled Twig templates
Database record that governs the Last-Modified is updated
Page is rendered with the old Twig template being used, goes into downstream cache
New version of the application image is deployed.

So the deployment timestamp must be the actual deployment time, not the container build timestamp.

However, that’s just a small side aspect that we will be able to solve.

Regarding @stof, you’re right with event listeners. Maybe they would need very high and/or low priorities, but that should work.

Regarding ETags, they're opaque identifiers and I am not sure I would mess around with them. But that also means the issue would not be addressed if you are used ETag-based validation.

nicolas-grekas · 2019-04-08T15:20:17Z

If the application is the same (ie built_time), then can't you suppose the cached items are the same also, within the boundaries of their own expiration dates?

mpdude · 2019-04-08T15:45:24Z

Yes, you can (I would).

The edge case is if you build the new version, but then serve a response with the old one and have a Last-Modified after the build time. If you deploy only then, the code above would not change anything.

(My assumption is that DI container build and cache warming happen on a CI and go into a Docker image, which is deployed sometime later. This might be totally wrong.)

nicolas-grekas · 2019-04-09T12:44:53Z

anything to do in core? shall we close?

mpdude · 2019-04-09T13:50:16Z

I have heard no objections so far, which I would like to interpret as that you would not reject such a PR as being inappropriate (out of scope for the core).

We could do this with event subscribers, thus keeping the Kernel simple. I think I could also make this work for ETags. And the whole thing (fix) could be activated with a framework configuration switch.

Ping @aschempp and @Toflar since I know you guys have dealt with the HttpKernel and caching a lot. So maybe you have an opinion on this as well.

Otherwise, feel free to close and I'll follow up with a PR.

fabpot · 2019-04-09T13:54:56Z

Reading this issue, I'm not this belongs to core. I can see why it could be useful, but this is somewhat opinionated and it would make for a great bundle.

mpdude · 2019-04-09T14:08:30Z

Ok, I can understand that – closing for now.

Toflar · 2019-04-09T14:13:16Z

I've quickly discussed this with @aschempp and to be honest, I don't think this is something for core either. Caching is such a complex topic and I don't think there's a one-size-fits-all solution here.
I suggest you try to find a solution for your use case first and then we might continue the discussion over at https://github.com/FriendsOfSymfony/FOSHttpCache which - in case we find something universally applicable - would be the suitable place to include such a feature. :)
Feel free to open the issue anyway in case you want to have more feedback before you try to implement someting. I'm sure @dbu would also have some valuable input :)

dbu · 2019-04-09T15:23:38Z

agreed with toflar. sounds like something that could fit well into FOSHttpCacheBundle (FOSHttpCache is not about symfony listeners but about proxy clients - but as you say, purging downstream caches is painful and/or impossible). basically like cache busting with an asset cache, but for if-none-match / if-modified-since... would be interesting to discuss further in FOSHttpCacheBundle.

mpdude changed the title ~~[RFC][HttpKernel] Option to make downstream HTTP caches expire on deployments~~ [RFC][HttpKernel] Make downstream HTTP caches expire on deployments Apr 8, 2019

javiereguiluz added HttpKernel RFC RFC = Request For Comments (proposals about features that you want to be discussed) labels Apr 8, 2019

mpdude closed this as completed Apr 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[RFC][HttpKernel] Make downstream HTTP caches expire on deployments #31007

[RFC][HttpKernel] Make downstream HTTP caches expire on deployments #31007

mpdude commented Apr 8, 2019 •

edited

Loading

nicolas-grekas commented Apr 8, 2019

Uh oh!

mpdude commented Apr 8, 2019

Uh oh!

stof commented Apr 8, 2019

Uh oh!

javiereguiluz commented Apr 8, 2019

Uh oh!

mpdude commented Apr 8, 2019

Uh oh!

nicolas-grekas commented Apr 8, 2019

Uh oh!

mpdude commented Apr 8, 2019

Uh oh!

nicolas-grekas commented Apr 9, 2019

Uh oh!

mpdude commented Apr 9, 2019

Uh oh!

fabpot commented Apr 9, 2019

Uh oh!

mpdude commented Apr 9, 2019

Uh oh!

Toflar commented Apr 9, 2019

Uh oh!

dbu commented Apr 9, 2019

Uh oh!

Uh oh!

[RFC][HttpKernel] Make downstream HTTP caches expire on deployments #31007

[RFC][HttpKernel] Make downstream HTTP caches expire on deployments #31007

Comments

mpdude commented Apr 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update

nicolas-grekas commented Apr 8, 2019

Uh oh!

mpdude commented Apr 8, 2019

Uh oh!

stof commented Apr 8, 2019

Uh oh!

javiereguiluz commented Apr 8, 2019

Uh oh!

mpdude commented Apr 8, 2019

Uh oh!

nicolas-grekas commented Apr 8, 2019

Uh oh!

mpdude commented Apr 8, 2019

Uh oh!

nicolas-grekas commented Apr 9, 2019

Uh oh!

mpdude commented Apr 9, 2019

Uh oh!

fabpot commented Apr 9, 2019

Uh oh!

mpdude commented Apr 9, 2019

Uh oh!

Toflar commented Apr 9, 2019

Uh oh!

dbu commented Apr 9, 2019

Uh oh!

mpdude commented Apr 8, 2019 •

edited

Loading