Skip to content

[Cache] Allow colon in key key (with redis at least) #45599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lyrixx opened this issue Mar 1, 2022 · 25 comments · Fixed by #59813
Closed

[Cache] Allow colon in key key (with redis at least) #45599

lyrixx opened this issue Mar 1, 2022 · 25 comments · Fixed by #59813

Comments

@lyrixx
Copy link
Member

lyrixx commented Mar 1, 2022

Description

I noticed something strange with symfony cache. It forbids to use a colon : in the key name.

But, namespacing keys with redis is (was?) a best practice, and so we lose some power when using redis insight for example.

In insight, I can profile redis, and it shows me what keys I use the most by grouping keys by pattern (a:b:c:*)

image

As you can see, the "TOP KEY PATTERNS" is poorly detailed.

  1. Redis recommends to namespace key with a colon
  2. Redis insight uses this pattern to analyze keys and group results
  3. Symfony forbit me to use a colon in the but
  4. And moreover, Symfony itself, in the very end does use a colon (between the app prefix seed and my key (f-yOWMrlsA:*on the screenshot)

So to me, we should simply remove this restriction

Note: psr restricts this usage

Key - [...] The following characters are reserved for future extensions and MUST NOT be supported by implementing libraries: {}()/@:

But I don't use PSR, And I personally don't care about this PSR since I'm using Symfony contracts.

Example

No response

@nicolas-grekas
Copy link
Member

personally don't care about this PSR since I'm using Symfony contracts

I'd be fine extending the range of allowed characters when the Symfony contracts are used.
Up for a PR doing so?

@carsonbot
Copy link

Thank you for this issue.
There has not been a lot of activity here for a while. Has this been resolved?

@xabbuh
Copy link
Member

xabbuh commented Aug 31, 2022

I fail to see what we could do here if PSR-6/PSR-16 are not changed.

@carsonbot carsonbot removed the Stalled label Aug 31, 2022
@lyrixx
Copy link
Member Author

lyrixx commented Sep 1, 2022

We can skip this recommendation? 🤷🏼

@lyrixx
Copy link
Member Author

lyrixx commented Sep 13, 2022

BTW, We already broke this recommendation. We use a colon between the namespace (prefix seed), and the real key. so I'm gonna do a PR to remove this useless limitation.

@nicolas-grekas
Copy link
Member

We did not break anything, the colon we use is not on a PSR-6 boundary so the spec doesn't apply there. It's actually a good thing since it means public keys cannot mess up with internal (prefixed) keys.

@nicolas-grekas
Copy link
Member

What about my proposal above? We can change our contracts.

@lyrixx
Copy link
Member Author

lyrixx commented Sep 13, 2022

yes, that's what I did 👍

@upyx
Copy link
Contributor

upyx commented Sep 18, 2022

It is not a good idea. It will break some implementations at least.

Cache adapters can use any characters that are supported by their backends. If some backend does not support some character, the adapter has to deal with it in some way. So it is practical when the caching interface supports only a subset of common characters.

Some adapters use reserved characters to replace unsupported characters on the backend (e.g. MemcachedAdapter), others use escaping that increases a key size.

Namespaces are needed for preventing collisions of names. It is convenient to have a few adapters with different namespaces instead of one with long prefixed keys. The AbstractAdapter uses a colon exactly for this, and it's normal to use a colon in a namespace, but not in a key. Look at namespaces in PHP, they work the same.

This example:

$app = new RedisAdapter($client, 'application:module:component_one');
$sys = new RedisAdapter($client, 'system');

$app->hasItem('the_key'); // in Redis it would be "application:module:component_one:the_key"
$sys->hasItem('the_key'); // in Redis it would be "system:the_key"

is much better than next one:

$cache = new RedisAdapter($client);

$cache->hasItem('application:module:component_one:the_key');
$cache->hasItem('system:the_key');

In the second example, it is always needed to prefix keys.

Currently, namespaces can't contain a colon (it does CacheItem::validateKey($namespace))). And that restriction is really useless, and that would be good to change.

@lyrixx
Copy link
Member Author

lyrixx commented Sep 19, 2022

@upyx IIUC, it means I will have to create many pools with different namespaces. In one class of my project, I think I use at least 20 different prefix! More over, I also use some ID in a namespace, and so it's not possible to create every possible combinaison in the static (config.yml) configuration.

Some code to illustrate what I'm talking about:

    public function startCrawl(string $crawlId, array $config)
    {
        $this->redis()->hMSet("crawler:{$crawlId}:configuration", $config);

        // Some Gauges, values keep incrementing:

        $this->redis()->tsCreate("crawler:{$crawlId}:ts:g:crawlable");
        $this->redis()->tsAdd("crawler:{$crawlId}:ts:g:crawlable", null, $this->redis()->sCard("crawler:{$crawlId}:crawlable"));
        $this->redis()->tsCreate("crawler:{$crawlId}:ts:g:downloaded");
        $this->redis()->tsAdd("crawler:{$crawlId}:ts:g:downloaded", null, 0);
        $this->redis()->tsCreate("crawler:{$crawlId}:ts:g:analyzed");
        $this->redis()->tsAdd("crawler:{$crawlId}:ts:g:analyzed", null, 0);
        $this->redis()->tsCreate("crawler:{$crawlId}:ts:g:crawled");
        $this->redis()->tsAdd("crawler:{$crawlId}:ts:g:crawled", null, 0);

[...]

@upyx
Copy link
Contributor

upyx commented Sep 19, 2022

@lyrixx

Some code to illustrate what I'm talking about

Looks a bit creepy, do you know? 😂

So, you have nested namespaces. Let's look at this in different way:

    public function startCrawl(string $crawlId, array $config)
    {
        $redis = $this->redis("crawler:{$crawlId}");
        $redisTs = $redis->nest("ts:g");

        $redis->hMSet("configuration", $config);

        $redisTs->tsCreate("crawlable");
        $redisTs->tsAdd("crawlable", null, $redis->sCard("crawlable"));
        $redisTs->tsCreate("downloaded");
        $redisTs->tsAdd("downloaded", null, 0);
        $redisTs->tsCreate("analyzed");
        $redisTs->tsAdd("analyzed", null, 0);
        $redisTs->tsCreate("crawled");
        $redisTs->tsAdd("crawled", null, 0);

        // other
    }

A bit clearer, isn't it?

The Symfony Cache does not help here. And changing the interface will not help too.

it's not possible to create every possible combinaison in the static (config.yml) configuration

Yes. A factory that will create pools on the fly is needed.

I imagine it like nest() method on pool:

framework:
  cache:
    crawler: '%env(REDIS_DSN)%'
class Crawler {
    private CacheInterface $cache;
    private array $config;

    public function __construct(CacheInterface $cacheCrawler, string $crawlId, array $config)
    {
        $this->cache = $cacheCrawler->nest($crawlId);
        $this->config = $config;
    }

    public function start()
    {
        $tsg = $this->cache->nest("ts:g");
        $etc = $this->cache->nest("some:other:etc");

        // ...
    }
}

Something like this... It's just an idea.

@nicolas-grekas
Copy link
Member

I like $pool->nest().

Note that ProxyAdapter can be used for this use case:
$subPool = new ProxyAdapter($parentPool, 'sub-namespace');

But having a dedicated method + interface could be nice.

@lyrixx
Copy link
Member Author

lyrixx commented Nov 8, 2022

Even if ->nest or other workarround are fine to me, I really disagree with

Looks a bit creepy, do you know? joy

It is basic string concatenation... I use this everywhere! I understand some people might find this ugly, but this is not. So IMHO this should be OK to compose manually the key and avoid manipulating too much object to achieve a very simple goal!

@upyx
Copy link
Contributor

upyx commented Nov 9, 2022

It is basic string concatenation... I use this everywhere! I understand some people might find this ugly, but this is not. So IMHO this should be OK to compose manually the key and avoid manipulating too much object to achieve a very simple goal!

Oh, nothing wrong with concatenation! I talked about repeating and breaking the DRY principle which is not important to the example. I'm very sorry that it touched you.

@carsonbot
Copy link

Thank you for this issue.
There has not been a lot of activity here for a while. Has this been resolved?

@carsonbot
Copy link

Could I get an answer? If I do not hear anything I will assume this issue is resolved or abandoned. Please get back to me <3

@lyrixx
Copy link
Member Author

lyrixx commented Jul 4, 2023

Keep it open

@lyrixx
Copy link
Member Author

lyrixx commented Jan 29, 2024

FTR, the PR on php-fig/fig-standards has been refused

@alfredbez
Copy link

alfredbez commented Jul 18, 2024

I also stumbled upon this issue in our project because we wanted to enable assertions in development and were made aware of this issue in our codebase. It's way easier to navigate the data when using colons, e.g. in Redis Commander:
image

Currently this hinders us from enabling assertions, see:

@upyx
Copy link
Contributor

upyx commented Jul 24, 2024

It's way easier to navigate the data when using colons

Colons are dedicated to namespace separator. So you can add namespaces to cache providers, and they will be saparated from keys by colons.

A namespace can be set directly to RedisAdapter or nest by ProxyAdapter (as Nikolas mentioned).

There are three typical ways to do so:

  1. In config. Namespaces can be set to static values as described in Symfony's documentation.
  2. By factory. You can write your own caching service factory with any logic you want.
  3. Inline. It's possible to wrap a cache service in ProxyAdapter right before using it (as in Nikolas' example above).

@alexandre-le-borgne
Copy link

alexandre-le-borgne commented Dec 4, 2024

I like $pool->nest().

Note that ProxyAdapter can be used for this use case: $subPool = new ProxyAdapter($parentPool, 'sub-namespace');

But having a dedicated method + interface could be nice.

Does not work, the key of the cache item is appended directly to the sub-namespace without using the ':' separator.

image
image

@nicolas-grekas
Copy link
Member

Can you try adding the colon to the namespace? 'sub-namespace:'

@alexandre-le-borgne
Copy link

alexandre-le-borgne commented Dec 4, 2024

@nicolas-grekas
Add ":" manually to the end of the sub-namespace and you will get "Cache key "establishments:" contains reserved characters "{}()/\@:"."

image

PS: I'm in symfony 6.4.16

@nicolas-grekas
Copy link
Member

Oh, then this should maybe be relaxed, this constructor argument doesn't have to adhere to PSR-6 rules.

@borjaberrocal87
Copy link

@nicolas-grekas

Hi! When do you think we will have ":" available as an available character?
Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment