Skip to content

[Cache] Advanced Memcached Adapter #20863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

robfrawley
Copy link
Contributor

@robfrawley robfrawley commented Dec 10, 2016

Q A
Branch? master
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes
Related Issues symfony/symfony-docs#7256
License MIT
Doc PR symfony/symfony-docs#7265
Related PRs #20858, #20752

Expansion of simple adapter from #20858. Replacement for #20752.

@robfrawley robfrawley changed the title Feature advanced memcached cache pool [Cache] Advanced Memcached Adapter Dec 10, 2016
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 10, 2016

@nicolas-grekas @stof This PR vastly cuts down on the server/option logic from prior proposals. It builds on #20858 and replaces the original proposal #20752.

Removed support for memcache extension as I agree with the sentiment expressed by @nicolas-grekas on its lack of active development and current feature-set.

@nicolas-grekas nicolas-grekas added this to the 3.3 milestone Dec 12, 2016
@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch 2 times, most recently from 8d1ff7c to 2da89b5 Compare December 12, 2016 16:19
@robfrawley
Copy link
Contributor Author

This has been updated with any changes made against #20858


private function setOption($opt, $val)
{
$toRestore = error_reporting(~E_ALL);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why changing the error reporting ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To return a false instead of an E_WARNING on invalid option passed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, IMO, the warning would be much easier to debug, as it explains what is wrong. So I'm -1 on this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://github.com/symfony/symfony/pull/20863/files#diff-ed071d63655500d46a7c7866cdb74cddR193 for test checking a false return on error instead of an E_WARNING. Is this something you don't think we should do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nicolas-grekas @stof Is the consensus to allow the E_WARNING and not handle a bool return here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HHVM simply ignores invalid options silently anyway, so the options are:

  • catch error and return bool on PHP
  • always return true on HHVM

OR if we don't catch the E_WARNING

  • throw E_WARNING on PHP
  • again, silent on HHVM

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OR, another option; continue catching the E_WARNNG but transform into an exception, which IMHO, may be the most developer friendly option? Does the cache component require the symfony polyfills? So we'd have access to get_last_error?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we could just let the error handler do it, without doing any special handling for E_WARNING happening inside setOption

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed error suppression and amended test to check for E_WARNING in latest push.

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch from 2da89b5 to 2f3aeed Compare December 12, 2016 16:51
*
* @return bool
*/
public function setup(array $dsns = array(), array $opts = array())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should expose such public API

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a middle-ground approach to allow a simple string DSN for create as @nicolas-grekas wanted, but also allow for an array of DSNs using a calls during service construction to this method.

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As well, without this method, all the logic from setup(), setOption(), resolveOption(), addServer(), parseServerDSN(), and isServerInClientPool() would need to be shoved into create(). I vastly prefer small, simple methods to a behemoth like RedisAdapter::createConnection(), IMHO.

* @param string[] $dsns
* @param mixed[] $opts
*
* @return bool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning a boolean does not make sense here:

  • handling errors by boolean return values is not user-friendly (no message explaining the issue)
  • it is ignored by create() anyway, meaning you don't have error handling at all

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returns $this now in latest push.

return array_values($srv);
}

protected function isServerInClientPool($host, $port)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why protected ?

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oversight from when this was part of a shared method for \Memcached and \Memcache implementations. No longer the case, obviously. Fixed in latest push.

protected function isServerInClientPool($host, $port)
{
return (bool) array_filter($this->client->getServerList(), function ($srv) use ($host, $port) {
return $host === array_shift($srv) && $port === array_shift($srv);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should avoid shifting the array. Mutating an array for reading it is a bad idea: it forces PHP to copy it (not that bad here as it has only a few items, but very bad in general

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone


protected function isServerInClientPool($host, $port)
{
return (bool) array_filter($this->client->getServerList(), function ($srv) use ($host, $port) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using array_filter is a bad idea: it forces to compare all servers even if the first one is a match, meaning that you force to always reach the worse case (when there is no match, we are indeed forced to compare them all)

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to foreach in latest push.

public static function setupBeforeClass()
{
if (!MemcachedAdapter::isSupported()) {
self::markTestSkipped('Extension memcached >2.1.0 required.');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the min version is 2.2.0 in the code, not 2.1.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

>2.1.0 and >=2.2.0 are the same thing, but I follow the logic of clarifying this. 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not the same. 2.2.0RC1 is > 2.1.0 but is not >= 2.2.0 (the same would be true for 2.1.1 if there was such a release)

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified in latest push.


public static function tearDownAfterClass()
{
self::$client->flush();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it be flushed between tests instead of at the end of the testcase, to ensure test isolation ?

I would even go further and avoid making the client static, ensuring we don't reuse it between tests at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how the Redis adapter handles it, but yeah; makes sense. Changed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The base test class already calls clear on the pool, that's why it's not needed, isn't it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you're right! Reverted to:

    public static function setupBeforeClass()
    {
        if (!MemcachedAdapter::isSupported()) {
            self::markTestSkipped('Extension memcached >=2.2.0 required.');
        }

        static::$client = static::getClientFromAdapter(MemcachedAdapter::create(
            static::defaultClientServer(),
            static::defaultClientOptions()
        ));

        parent::setupBeforeClass();
    }

return new MemcachedAdapter(self::$client, str_replace('\\', '.', __CLASS__), $defaultLifetime);
}

public function testCreateAdaptor()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it be adapter ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup ;-) But that test is actually gone now.

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch 2 times, most recently from 3947b38 to f780643 Compare December 12, 2016 17:11
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 12, 2016

Latest commits bring in line with latest changes to #20858 (will squash shortly)

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch 2 times, most recently from c46a207 to 762e10d Compare December 12, 2016 18:09
@robfrawley
Copy link
Contributor Author

Removed all instanced of array_shift (there were three)...

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch from 762e10d to 2c83d2c Compare December 12, 2016 18:55
@robfrawley
Copy link
Contributor Author

Squashed commits. All changes from #20858 implemented in this PR.

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch 3 times, most recently from 904aa74 to b2b131c Compare December 12, 2016 21:17
fabpot added a commit that referenced this pull request Dec 13, 2016
This PR was merged into the 3.3-dev branch.

Discussion
----------

[Cache] Simple Memcached Adapter

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | yes
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets |
| License       | MIT
| Doc PR        |
| Related PRs | #20863, ~~#20752~~

Commits
-------

12de2ae memcached cache pool adapter
@nicolas-grekas
Copy link
Member

Rebase needed

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch from b2b131c to 43a4559 Compare December 13, 2016 17:29
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 13, 2016

Done!

@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 13, 2016

Also, amended this commit to be in-line with @fabpot style fixes from 51d13cc against my first Memcached PR #20858

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch from 43a4559 to 25590a9 Compare December 13, 2016 17:37
@nicolas-grekas
Copy link
Member

Let's go for string or array of DSN.
But really, we only need one public method. We don't need nor want a single more.
You have to consider that each new public/protected/interface creates a maintainability burden on the community to keep the BC promise.
In this case, all the other added methods are just helpers. There's no way we're going to want to maintain these.

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch 2 times, most recently from e6baecc to 23fbf08 Compare December 15, 2016 20:46
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 15, 2016

Removed the other interface methods, except for isSupported() which IMO seems like a much more applicable place to have this check (in the Client class, such as the MemcacheClient) than within the adapter, and is likely applicable to all adapter clients that'll be written (such as RedisClient), right? If not, give me a heads up and I'll get rid of it.

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch from 23fbf08 to fa4def0 Compare December 15, 2016 20:52
@nicolas-grekas
Copy link
Member

Yes, please remove AbstractClient and the interface. Please also make sure that the createClient (or createConnection) returns a Memcached instance.
Wrapping it in a "new static" reduces the usefulness of the method.

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch from fa4def0 to d35b594 Compare December 15, 2016 21:24
@robfrawley
Copy link
Contributor Author

Makes sense (re: \Memcached return). Interface/abstract removed as well!

Copy link
Member

@nicolas-grekas nicolas-grekas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only need one single new public method in the whole PR :)

/**
* @author Rob Frawley 2nd <rmf@src.run>
*/
class MemcachedClient
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class should be removed: creating a Memcached layer is not the responsibility of the Cache component. But by having this class, people will use it as such. An alternative could be to mark it as internal, but I'm not sure this makes more sense.

Copy link
Contributor Author

@robfrawley robfrawley Dec 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marked @internal and final for now. I really, really, really want to avoid shoving all this logic into a single createConnection method within the Adapter itself...

*/
class MemcachedClient
{
private static $serverDefaults = array(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need server defaults in the cache component (we may have in FrameworkBundle later on, but that's another topic.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the distribution defaults for an installed memcached server with no user configuration applied. Additionally, this doesn't actually add a default server if no DSN is passed. At a minimum, the user still needs to pass memcached: as the DSN. An empty array or empty string will not add a server.


private static $optionDefaults = array(
'compression' => true,
'libketama_compatible' => true,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about enabling binary mode by default?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost added it in my first pass actually, but decided not to be presumptuous. Since you feel similarly, I say yes.

));
}

private function resolveServerReturn($server)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about username+ password for sasl?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed for both host and socket DSNs:

memcached://user:password@127.0.0.1
memcached://user:password@/var/local/run/memcached.socket

$this->client->flushBuffers();
}

parent::__destruct();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the parent be called before?

public function __destruct()
{
if (!$this->client->isPersistent() && method_exists($this->client, 'flushBuffers')) {
$this->client->flushBuffers();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not done automatically when the connection is destructed?

Copy link
Contributor Author

@robfrawley robfrawley Dec 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, this method isn't even documented yet, found it when looking through the extension source to determine available options for the documentation PR. It seems like this is supposed to be called if the new buffer_writes option is enabled. But I'd have to take a closer look at the source of the extension to confirm if/when it is called by the extension itself. As far as I can tell via a first pass, it is only exposing the function from the library itself and not utilizing it within the extension otherwise.

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch from d35b594 to 315bdb5 Compare December 16, 2016 05:05
@robfrawley
Copy link
Contributor Author

Also, made the MemcachedClient final, as well as @internal

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch 4 times, most recently from 444f9fc to 1fc814d Compare December 16, 2016 05:29
@robfrawley robfrawley changed the title [Cache] [WIP] Advanced Memcached Adapter [Cache] Advanced Memcached Adapter Dec 17, 2016
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 17, 2016

Documentation PR submitted (assuming this is merged as-is) as symfony/symfony-docs#7265: view the resulting output on platform.sh.

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch 4 times, most recently from 6e780c3 to acea4ad Compare December 17, 2016 20:21
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 17, 2016

@robfrawley robfrawley force-pushed the feature-advanced-memcached-cache-pool branch from acea4ad to c55532c Compare December 24, 2016 20:12
@robfrawley
Copy link
Contributor Author

@nicolas-grekas Do you have any idea why AppVeyor is erroring at seemingly random places in the test suite: on line 259 and on line 999?

Also, what more needs to be done prior to merging this?

@nicolas-grekas
Copy link
Member

Continued in #21108
Thank you @robfrawley!

nicolas-grekas added a commit that referenced this pull request Jan 4, 2017
… to MemcachedAdapter (nicolas-grekas, robfrawley)

This PR was merged into the 3.3-dev branch.

Discussion
----------

[Cache] Add DSN, createClient & better error reporting to MemcachedAdapter

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | yes
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets | -
| License       | MIT
| Doc PR        | symfony/symfony-docs#7265

Replaces #20863
ping @robfrawley: would you mind opening a doc PR for this?

Commits
-------

87030b4 [cache] Add tests for MemcachedAdapter::createClient()
e109438 [Cache] Add DSN, createClient & better error reporting to MemcachedAdapter
xabbuh added a commit to symfony/symfony-docs that referenced this pull request Mar 13, 2017
…ctoring (robfrawley, javiereguiluz)

This PR was merged into the master branch.

Discussion
----------

[Cache] Memcached Adapter Docs and Cache Pool Docs Refactoring

Documentation for symfony/symfony#20863 and symfony/symfony#20858. PR symfony/symfony#20863 is still being written/finalized, so this is a WIP following that PR at this time. Resolves #7256.

Commits
-------

7db929c corrected spelling/grammar
c49d427 cleanup all cache adapter documentation, extend redis docs
e3eff5b Reviewed the entire article
d4292b7 refactor cache pool documentation and add memcached adapter docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants