Skip to content

[HttpKernel] Add $kernel->getBuildDir() to separate it from the cache directory #36515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 21, 2020

Conversation

mnapoli
Copy link
Contributor

@mnapoli mnapoli commented Apr 21, 2020

Q A
Branch? master
Bug fix? no
New feature? yes
Deprecations? no
Tickets Fix #23354
License MIT
Doc PR symfony/symfony-docs#...

In order to support deploying on read-only filesystems (e.g. AWS Lambda in my case), I have started implementing #23354.

This introduces $kernel->getBuildDir():

  • $kernel->getBuildDir(): for cache that can be warmed and deployed as read-only (compiled container, annotations, etc.)
  • $kernel->getCacheDir(): for cache that can be written at runtime (e.g. cache pools, session, profiler, etc.)

I have probably missed some places or some behavior of Symfony that I don't know. Don't consider this PR perfect, but rather I want to help move things forward :)

TODO:

  • Changelog
  • Upgrade guide
  • Documentation

public function getTmpDir()
{
// Returns $this->getCacheDir() for backward compatibility
return $this->getCacheDir();
Copy link
Contributor Author

@mnapoli mnapoli Apr 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the core of the PR :)

For backward compatibility, I return the current cache directory.

In future versions, this could be for example var/build. In the meantime, users can override this method to set it to a different path, for example /tmp.

@mnapoli mnapoli force-pushed the tmp-dir branch 2 times, most recently from 61bd29e to fee57fb Compare April 21, 2020 12:40
@nicolas-grekas nicolas-grekas changed the title #23354 Add $kernel->getTmpDir() to separate it from the cache directory [HttpKernel] Add $kernel->getTmpDir() to separate it from the cache directory Apr 21, 2020
@nicolas-grekas nicolas-grekas added this to the next milestone Apr 21, 2020
@javiereguiluz
Copy link
Member

This solution probably makes a lot of sense ... but at first seems confusing:

Problem:

  • cache/ is read+write, my server is read only

Solution:

  • let's make cache/ read-only
  • let's create a new tmp/ dir which is read+write

@mnapoli
Copy link
Contributor Author

mnapoli commented Apr 22, 2020

@javiereguiluz yes, this was brought up in #23354 (comment) and I think that's a valid alternative: keep cache as-is, introduce a new build directory.

I'm waiting for feedback on this new approach, I started looking into it a pull request as well but haven't opened it yet.

@fabpot
Copy link
Member

fabpot commented Aug 18, 2020

This is a good move IMHO.

Reading the related issue, I think we should get the naming right. Keeping the cache/ directory as is and creating a new one for the "static"/"build" cache seems better to me. Let's name it build/ for now (I don't have any other name in mind).

@mnapoli mnapoli changed the title [HttpKernel] Add $kernel->getTmpDir() to separate it from the cache directory [HttpKernel] Add $kernel->getBuildDir() to separate it from the cache directory Aug 19, 2020
@mnapoli
Copy link
Contributor Author

mnapoli commented Aug 19, 2020

Great! I have updated the PR.

Note that after struggling for a while I did not succeed running the tests locally (Error : Call to undefined method Symfony\Bundle\FrameworkBundle\Tests\Console\ApplicationTest::assertMatchesRegularExpression()), so I'm kind of in the blind here. I've pushed what I have done, we'll see on Travis.

Review is welcome!

Copy link
Member

@fabpot fabpot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (minor comment)

WDYT @symfony/mergers?

@stof
Copy link
Member

stof commented Aug 20, 2020

@mnapoli are you using the ./phpunit script to run tests ?

@weaverryan
Copy link
Member

This makes a lot of sense to me. And the fact that the new getBuildDir() returns getCacheDir() means that we're adding this flexibility, but at zero "learning about some new change" cost to existing users that won't care about this.

Very nice!

if (!is_writable($realCacheDir)) {
throw new RuntimeException(sprintf('Unable to write in the "%s" directory.', $realCacheDir));
}

$io->comment(sprintf('Clearing the cache for the <info>%s</info> environment with debug <info>%s</info>', $kernel->getEnvironment(), var_export($kernel->isDebug(), true)));
$this->cacheClearer->clear($realBuildDir);
Copy link
Contributor

@ogizanagi ogizanagi Aug 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this command avoid removing the build dir by default? What about adding a --build option?
For the ones not overriding KernelInterface::getBuildDir(), this option won't do anything more.
But for the ones overriding, you might want to call this command to clear the read-write cache, but not the read-only build dir, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a first step, we must clear everything like today.

@mnapoli
Copy link
Contributor Author

mnapoli commented Aug 20, 2020

@stof thanks for the tip, I didn't. Still, when I use it the tests crash with:

PHP Fatal error: Uncaught Error: Call to a member function stop() on null in [...]/src/Symfony/Contracts/HttpClient/Test/TestHttpServer.php:27

Anyway, I don't really want to pollute the discussion with this.

@fabpot
Copy link
Member

fabpot commented Aug 21, 2020

Thank you @mnapoli.

@nicolas-grekas nicolas-grekas modified the milestones: next, 5.2 Oct 5, 2020
@fabpot fabpot mentioned this pull request Oct 5, 2020
chalasr added a commit that referenced this pull request Dec 15, 2020
This PR was merged into the 5.2 branch.

Discussion
----------

[FrameworkBundler] Fix cache:clear with buildDir

| Q             | A
| ------------- | ---
| Branch?       | 5.2
| Bug fix?      | yes
| New feature?  | no
| Deprecations? | no
| Tickets       | Fix #39232
| License       | MIT
| Doc PR        |

Since #36515 there are 2 caches dir `cacheDir` and `buildDir`. For BC reason both points to the same folders, but when app don't use the same folder, many thing are broken:
This PR fixes several issues introduces by the above PR:
- some files are persisted in the wrong folder (`App_KernelDevDebugContainerDeprecations.log`, `App_KernelDevDebugContainer.xml`)
- LoggerDataCollector looks into cache_dir, while `Compiler.log` is written in build_dir and `Deprecations.log` were written in cache_dir before this PR
- the logic that mirror cacheDir into buildDir at the end of CacheClearCommand does not make sens when `cache_dir` and `build_dir` are not identical.
- Files generated in cacheDir are trashed at the end of CacheWarming (initial issue)

Commits
-------

ea68966 Fix cache:clear with buildDir
javiereguiluz added a commit to EasyCorp/EasyAdminBundle that referenced this pull request Nov 27, 2024
This PR was squashed before being merged into the 4.x branch.

Discussion
----------

Use buildDir for cache

Feature available since Symfony 5.2 - symfony/symfony#36515

Commits
-------

9daaade Use buildDir for cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add var/tmp / $kernel->getTmpDir()