Skip to content

Memcached session handler issue with clock skew #48619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tvlooy opened this issue Dec 12, 2022 · 6 comments
Closed

Memcached session handler issue with clock skew #48619

tvlooy opened this issue Dec 12, 2022 · 6 comments

Comments

@tvlooy
Copy link
Contributor

tvlooy commented Dec 12, 2022

Symfony version(s) affected

All versions

Description

In 0216e05 the TTL values switch from relative time to absolute timestamps. I wonder why this change was made.

"Memcached implements a monotonic clock to ensure it always goes forward generally when time goes forward for the computer."
memcached/memcached#278

But the internal clock can have skew in some environments. I encountered a Debian system running on VMware ESX host that actually has this. So while the OS clock is correct, the internal memcached clock is not (can be checked with the memcached stats command).

If the time drifts too far, memcached will see that entries are expired at the point you are adding them, and not add them to the store. So, your sessions are not created, logins will fail, carts can not be created, ...

As suggested here memcached/memcached#486 it "relative expiration will still work as advertised since the drift is slow".

So while this seems like an edge case and maybe should be checked on the the infrastructure level instead of in this code, it does feel like this would fix it in all situations.

How to reproduce

Only reproducible on a system that has clock skew on the memcached monotonic timer.

$ date
Mon 12 Dec 2022 08:18:37 PM CET

$ echo stats | nc 127.0.0.1 11211 | grep time
STAT time 1670876432

$ date -d @1670876432
Mon 12 Dec 2022 09:20:32 PM CET

Possible Solution

If the time() calls are removed from the MemcachedSessionHandler that would result in relative TTL's and clock skew will no longer be an issue. I'll do a MR if we agree to fix this.

Additional Context

No response

@derrabus
Copy link
Member

In 0216e05 the TTL values switch from relative time to absolute timestamps. I wonder why this change was made.

At first, your issue description sounded to me like a recent change broke your application. For the record: That commit you're talking about is ten years old. 😅

@tvlooy
Copy link
Contributor Author

tvlooy commented Dec 12, 2022

At first, your issue description sounded to me like a recent change broke your application. For the record: That commit you're talking about is ten years old. sweat_smile

Yes, I am aware of that. 😂 I do think it's a bit an edge case, but I think switching to relative TTL would fix it in all circumstances. I also can't comment on how much the memcached session handler is actually being used, ...

@derrabus
Copy link
Member

Yeah, I think you have a point. The current code requires the clocks of your webserver and your memcached server to be in sync, which is close to impossible to ensure unless they're the same machine.

@nicolas-grekas
Copy link
Member

nicolas-grekas commented Dec 13, 2022

How can this be a problem in real world use case? The drift is slow and the expiration date of sessions is far away in the future (weeks if not months.) Even a few hours of drifts shouldn't make a difference, isn't it?
I can't tell why this went from relative to absolute though. Maybe @vicb knows since he authored #4227? (but 10 years ago... :) )

@tvlooy
Copy link
Contributor Author

tvlooy commented Dec 13, 2022

In Shopware e-commerce the session TTL is only 1 hour by default for visitors. The clock of memcached keeps going too fast, while the OS clock is being corrected. So after a week or so, memcached is living an hour in the future. If an entry is added that only has 1 hour TTL it's expired when it is being added and memcached will just not add it anymore.
This is actually a real world problem I encounter (luckily not in production). I can find various VMware reports of people that say that the guest clock running too fast under ESX. So this must be a thing other people experience too.

@nicolas-grekas
Copy link
Member

Could you submit a PR to do relative TTLs?

nicolas-grekas added a commit that referenced this issue Dec 14, 2022
…sionHandler (tvlooy)

This PR was merged into the 5.4 branch.

Discussion
----------

[HttpFoundation] Use relative timestamps with MemcachedSessionHandler

| Q             | A
| ------------- | ---
| Branch?       | 5.4
| Bug fix?      | yes
| New feature?  |
| Deprecations? |
| Tickets       | Fix #48619
| License       | MIT
| Doc PR        |

Use relative timestamps. If for some reason your client is not in sync with the server clock, you at least agree about the relative TTL. Or see #48619 for a case where the memcached monotonic clock is going faster than the server and causes users to add entries that are expired on entry.

Commits
-------

cb2178d Use relative timestamps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants