Skip to content

[Cache] Simple Memcached Adapter #20858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 13, 2016

Conversation

robfrawley
Copy link
Contributor

@robfrawley robfrawley commented Dec 10, 2016

Q A
Branch? master
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes
Fixed tickets
License MIT
Doc PR symfony/symfony-docs#7265
Related PRs #20863, #20752

@robfrawley robfrawley changed the title simple memcached cache pool adapter [Cache] Simple memcached cache pool adapter Dec 10, 2016
@robfrawley robfrawley changed the title [Cache] Simple memcached cache pool adapter [Cache] Simple Memcached Adapter Dec 10, 2016
@robfrawley robfrawley force-pushed the feature-simple-memcached-cache-pool branch 4 times, most recently from c5ed119 to 88b9ab9 Compare December 10, 2016 17:44
protected $skippedTests = array(
//'testExpiration' => 'Testing expiration slows down the test suite',
//'testHasItemReturnsFalseWhenDeferredItemIsExpired' => 'Testing expiration slows down the test suite',
//'testDefaultLifeTime' => 'Testing expiration slows down the test suite',
Copy link
Contributor Author

@robfrawley robfrawley Dec 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These commented-out lines should be removed prior to merge.

@robfrawley robfrawley force-pushed the feature-simple-memcached-cache-pool branch from e0d09f5 to 740bce0 Compare December 10, 2016 18:05

public static function setupBeforeClass()
{
if (!extension_loaded('memcached') || !version_compare(phpversion('memcached'), '2.1.0', '>')) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only 2.2.x and 3.x are supported. Version 2.1.x causes symbol error in extension when requesting last result code.

Copy link
Contributor Author

@robfrawley robfrawley Dec 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://travis-ci.org/symfony/symfony/jobs/182877309#L2410 for example of the above described issue. Only applicable to memcached version 2.1.0 in Travis PHP 5.5 environment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To the unaware reader (like me a few minutes ago), this reads as if 2.1.1 is fine. But there is no 2.1.x, x>0
Could the check be ">=2.2.0"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, this check could be moved to a public static function isSupported() on the adapter (see ApcuAdapter for similar concern)


private function getIdsByPrefix($namespace)
{
if (false === $ids = $this->client->getAllKeys()) {
Copy link
Member

@nicolas-grekas nicolas-grekas Dec 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering: how does this method play with Memcached::OPT_PREFIX_KEY? Does it ignore it (ie return all keys, even those that don't start with the prefix) or does it filter by prefix?

Copy link
Contributor Author

@robfrawley robfrawley Dec 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the docs:

Memcached::getAllKeys() queries each memcache server and retrieves an array of all keys stored on them at that point in time [...]

So it sounds like it ignores that setting, but I haven't used Memcached::OPT_PREFIX_KEY myself, so I don't think we'd find a conclusive answer without delving into the C code for the extension.

Copy link
Contributor Author

@robfrawley robfrawley Dec 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: Remember, I don't have a combination of libmemcached/ext-memcached that actually supports this method so testing this behavior is...difficult. I'll see if I can't compile a working combination using phpenv later tonight.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should drop the use of getAllKeys and reduce doClear to return $this->client->flush();: you can't make it work, and it comes with many notices saying this can kill kittens.

*/
protected function doFetch(array $ids)
{
foreach ($this->client->getMulti($ids) as $id => $val) {
Copy link
Member

@nicolas-grekas nicolas-grekas Dec 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't we just return $this->client->getMulti($ids);?

*/
protected function doHave($id)
{
return $this->client->get($id) !== false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since false is a valid value, this can't work, isn't it?
I guess this should be

        $this->client->get($id);

        return $this->client->getResultCode() === \Memcached::RES_SUCCESS;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is an error on my part: intended for that to be an OR (||). The idea is to avoid the additional call to $this->client->getResultCode() if the $this->client->get($id) isn't false.

Copy link
Contributor Author

@robfrawley robfrawley Dec 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I meant to have there (and what I'd recommend we use):

protected function doHave($id)
    {
        return $this->client->get($id) !== false
            || $this->client->getResultCode() !== \Memcached::RES_NOTFOUND;
    }

Kind of a micro-optimization, but worth avoiding an additional function call if unnecessary, IMHO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated per above comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking against === RES_SUCCESS looks required to me, !== opens for all the other codes

Copy link
Contributor Author

@robfrawley robfrawley Dec 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that change does make sense. I missed the change from !== RES_NOTFOUND to === RES_SUCCESS in your comment. Updated in latest push.

Copy link
Member

@nicolas-grekas nicolas-grekas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're almost good for the first step :)

use Symfony\Component\Cache\Adapter\MemcachedAdapter;

/**
* @author Rob Frawley 2nd <rmf@src.run>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we usualy don't add author on tests (even if test some have, but they could be removed)


public static function setupBeforeClass()
{
if (!extension_loaded('memcached') || !version_compare(phpversion('memcached'), '2.1.0', '>')) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, this check could be moved to a public static function isSupported() on the adapter (see ApcuAdapter for similar concern)

}

/**
* @group memcachedAdapter
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't use groups like this

/**
* @group memcachedAdapter
*/
public function testAdapterAndClientCreationAndServers()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure this test is useful: it tests only test methods/behavior, I'd remove it

@robfrawley robfrawley force-pushed the feature-simple-memcached-cache-pool branch from 740bce0 to 1143944 Compare December 11, 2016 22:02
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 11, 2016

@nicolas-grekas New pushed code with changes per your comments (and in some cases, my responses).

Also, un-commented test skips at this point as prior tests have shown testExpiration, testHasItemReturnsFalseWhenDeferredItemIsExpired, and testDefaultLifeTime pass and I assume we want to skip those once this code is merged.

Otherwise, a bit of cleanup in the test class as well. Should be good to go @nicolas-grekas! Let me know if I can do anything else to get this merge ready.

@robfrawley robfrawley force-pushed the feature-simple-memcached-cache-pool branch 3 times, most recently from 3b2676d to 778d5e6 Compare December 11, 2016 23:05
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 12, 2016

@nicolas-grekas I am unable to find any combination of libmemcached and ext-memcached that result in a non-false return for \Memcached::getAllKeys().

How do you feel about replacing this class's getIdsByPrefix() with the following:

private function getIdsByPrefix($namespace)
{
    if (false === ($ids = $this->client->getAllKeys())
     && false === ($ids = $this->getAllKeysMemcache())) {
        return false;
    }

    return array_filter((array) $ids, function ($id) use ($namespace) {
        return 0 === strpos($id, $namespace);
    });
}

private function getAllKeysMemcache()
{
    static $client = null;

    if (false === $client
        || (null === $client
            && false === $client = $this->createMemcacheClient())) {
        return false;
    }

    $ids = array();
    foreach ($client->getExtendedStats('slabs') as $group) {
        foreach ($group as $slabId => $metadata) {
            if (!is_array($metadata)) {
                continue;
            }
            foreach ($client->getExtendedStats('cachedump', (int)$slabId, 1000) as $slabIds) {
                if (is_array($slabIds)) {
                    $ids = array_merge($ids, array_keys($slabIds));
                }
            }
        }
    }

    return $ids;
}

private function createMemcacheClient()
{
    if (!extension_loaded('memcache')) {
        return false;
    }

    $client = new \Memcache();
    foreach ($this->client->getServerList() as $srv) {
        $client->addServer(array_shift($srv), array_shift($srv));
    }

    return $client;
}

This ensures the check for the memcache extension fallback for obtaining all keys only occurs once, and then allows for proper, namespace-enabled flushing if that extension is available.... Thoughts?

Note: Mixing these extensions in terms of their values is dangerous, due to serialization differences, but in terms of simply returning keys is harmless.

Due to the fact that \Memcached::getAllKeys() seems perpetually broken on all major distributions/ext-versions I've tested, this seems like a possible functional (arguably hackie) solution...

Basic logic required for this implementation:

When ext-memcache Is Not Available

On First Call

  • Method call: getAllKeysMemcache() (final return bool:false)
  • Conditional check: false === $client
  • Conditional check: null === $client
  • Conditional check: false === $client = $this->createMemcacheClient())
  • Conditional check extension_loaded()

On Subsequent Calls

  • Method call: getAllKeysMemcache() (final return bool:false)
  • Conditional check: false === $client

When ext-memcache Is Available

On First Call

  • Method call: getAllKeysMemcache() (final return string[])
  • Conditional check: false === $client
  • Conditional check: null === $client
  • Conditional check: false === $client = $this->createMemcacheClient())
  • Conditional check extension_loaded()
  • Instantiate and add servers to \Memcache instance
  • Perform loop to get ids

On Subsequent Calls

  • Method call: getAllKeysMemcache() (final return string[])
  • Conditional check: false === $client
  • Perform loop to get ids

@nicolas-grekas
Copy link
Member

I think this is going too far, that we should give up and just call flush... :)

$this->client = $client;
}

public static function isSupported()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be moved as first method in the class (same as e.g. ApcuAdapter)

@nicolas-grekas nicolas-grekas added this to the 3.3 milestone Dec 12, 2016
@robfrawley robfrawley force-pushed the feature-simple-memcached-cache-pool branch from 778d5e6 to 64f3cce Compare December 12, 2016 15:43
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 12, 2016

Ha; fair enough. This should be good to go in that case. Updated per most recent comments.

return $isCleared;
}

private function getIdsByPrefix($namespace)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @robfrawley: :)

    protected function doClear($namespace)
    {
        return $this->client->flush();
    }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a commit with https://github.com/symfony/symfony/pull/20858/files#diff-e01499fab5ad7f5e36517d157bc13e2fR96 to check if any Travis environment actually has a functional combination of the extension and library. After checking https://travis-ci.org/symfony/symfony/builds/183319461, if none actually utilize this, then I'm with you 100%.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope. None call the extra logic: removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing that also raises coverage to 100%, so it's a solid move in my book.

@robfrawley robfrawley force-pushed the feature-simple-memcached-cache-pool branch from 0673846 to 94e8e88 Compare December 12, 2016 17:44
@robfrawley
Copy link
Contributor Author

robfrawley commented Dec 12, 2016

@nicolas-grekas Should I make the same setUp and tearDown changes @stof recommended for my other PR at #20863 (comment)

{
$toDelete = count($ids);
foreach ((array) $this->client->deleteMulti($ids) as $result) {
if (true === $result || \Memcached::RES_NOTFOUND === $result) {
Copy link
Member

@nicolas-grekas nicolas-grekas Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, only one remaining comment here: in which case can $result be true? when the cast is in effect and only then (look at the doc, I'd say yes?)

Shouldn't the implem be this one?

if (true === $results = $this->client->deleteMulti($ids)) {
    return true;
}
$toDelete = count($ids);
foreach ($results as $result) {
    if (\Memcached::RES_SUCCESS === $result || \Memcached::RES_NOTFOUND === $result) {
        --$toDelete;
    }
}

return 0 === $toDelete;

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return of $this->client->deleteMulti($ids) is actually always bool[] now that we have isSupported() require >=2.2.0.

So the cast can be removed now, to simply have

        $toDelete = count($ids);
        foreach ($this->client->deleteMulti($ids) as $result) {
            if (true === $result || \Memcached::RES_NOTFOUND === $result) {
                --$toDelete;
            }
        }

        return 0 === $toDelete;

That was to support <2.2.0 extension versions that would return a single bool

Copy link
Contributor Author

@robfrawley robfrawley Dec 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scratch that, you're right. The check can now simply be:

        $toDelete = count($ids);
        foreach ($this->client->deleteMulti($ids) as $result) {
            if (\Memcached::RES_SUCCESS === $result || \Memcached::RES_NOTFOUND === $result) {
                --$toDelete;
            }
        }

        return 0 === $toDelete;

The situation where it returns bool is behind us with the version requirement in isSupported().

@robfrawley robfrawley force-pushed the feature-simple-memcached-cache-pool branch from 94e8e88 to 12de2ae Compare December 12, 2016 18:09
@nicolas-grekas
Copy link
Member

Let's go :)
👍

@fabpot
Copy link
Member

fabpot commented Dec 13, 2016

Thank you @robfrawley.

@fabpot fabpot merged commit 12de2ae into symfony:master Dec 13, 2016
fabpot added a commit that referenced this pull request Dec 13, 2016
This PR was merged into the 3.3-dev branch.

Discussion
----------

[Cache] Simple Memcached Adapter

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | yes
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets |
| License       | MIT
| Doc PR        |
| Related PRs | #20863, ~~#20752~~

Commits
-------

12de2ae memcached cache pool adapter
xabbuh added a commit to symfony/symfony-docs that referenced this pull request Mar 13, 2017
…ctoring (robfrawley, javiereguiluz)

This PR was merged into the master branch.

Discussion
----------

[Cache] Memcached Adapter Docs and Cache Pool Docs Refactoring

Documentation for symfony/symfony#20863 and symfony/symfony#20858. PR symfony/symfony#20863 is still being written/finalized, so this is a WIP following that PR at this time. Resolves #7256.

Commits
-------

7db929c corrected spelling/grammar
c49d427 cleanup all cache adapter documentation, extend redis docs
e3eff5b Reviewed the entire article
d4292b7 refactor cache pool documentation and add memcached adapter docs
@fabpot fabpot mentioned this pull request May 1, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants