Skip to content

[Lock] Added Symfony/Component/Lock/Store/MongoDbStore #27346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 29 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
157fcd7
#27345 Added Symfony/Component/Lock/Store/MongoDbStore
May 23, 2018
e73f663
#27345 Fixed typo
May 23, 2018
3838412
#27345 fixed coding standards
May 23, 2018
1a660e6
#27345 Removed dev requirement for mongodb/mongodb
May 23, 2018
d65fddf
#27345 Removed unnessasary reduceLifetime
May 24, 2018
f3cc220
#27345 Improved documentation, exception handling, removed configurab…
May 31, 2018
ec8d217
#27345 fixed typo and @param indentation
May 31, 2018
a1a3be7
#27345 fixed @param indentation
May 31, 2018
1b150ba
#27345 moved clock discrepancy handling to a constructor option 'drift'
May 31, 2018
73bb802
#27345 fixed phpdoc spacing
May 31, 2018
5e86904
#27345 removed drift option in favour or recommending setting TTL higher
May 31, 2018
8572c21
#27345 Added public createTTLIndex method
Jun 5, 2018
dbdad36
#27345 Fixed coding standards
Jun 5, 2018
84c4d15
#27345 Fixed coding standards
Jun 5, 2018
d577191
#27345 Fixed coding standards
Jun 5, 2018
2ce68f5
#27345 Fixed coding standards
Jun 5, 2018
5446b11
#27345 Improved documentation
Jun 5, 2018
a9b85d6
#27345 Fixed spacing for fabbot
Jun 5, 2018
6fdc602
#27345 reordered documentation in order of importance
Jun 5, 2018
9d80a0d
#27345 Fixed spacing for fabbot
Jun 5, 2018
1ac702a
#27345 Fixed spacing for fabbot
Jun 5, 2018
4ba5b41
#27345 Updated return type and exception handling on createTTLIndex
Jun 6, 2018
58ef873
#27345 Updated exception handling on createTTLIndex
Jun 6, 2018
36c3af4
#27345 Fixed spacing for fabbot
Jun 6, 2018
168f194
#27345 Removed initialTtl from array and fixed unit test
Jun 6, 2018
abc72f8
#27345 phpdoc adjusted for fabbot
Jun 6, 2018
e062bf2
#27345 phpdoc adjusted for fabbot
Jun 6, 2018
f2e23b4
#27345 trigger ci retry
Jun 14, 2018
97e5711
#27345 trigger ci retry
Jun 14, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
#27345 Improved documentation, exception handling, removed configurab…
…le fields
  • Loading branch information
Joe Bennett committed May 31, 2018
commit f3cc220b70d9aed671b76a7f78e10f98a360bd5a
150 changes: 53 additions & 97 deletions src/Symfony/Component/Lock/Store/MongoDbStore.php
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
use Symfony\Component\Lock\Exception\InvalidArgumentException;
use Symfony\Component\Lock\Exception\LockConflictedException;
use Symfony\Component\Lock\Exception\LockExpiredException;
use Symfony\Component\Lock\Exception\LockStorageException;
use Symfony\Component\Lock\Exception\NotSupportedException;
use Symfony\Component\Lock\Key;
use Symfony\Component\Lock\StoreInterface;

Expand All @@ -31,34 +33,36 @@ class MongoDbStore implements StoreInterface

/**
* @param \MongoDB\Client $mongo
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to pass a dns for lazy connection? See PdoTrait from cache components?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A MongoDB\Client instance is already lazy. I can turn off my database and still:

$client = new \MongoDB\Client();
$store = new \Symfony\Component\Lock\Store\MongoDbStore($client, [
    'database' => 'test',
]);

only if I add something like:

$store->createTTLIndex();

does it throw an exception.

* @param array $options
* @param array $options
*
* database: The name of the database [required]
* collection: The name of the collection [default: lock]
* resource_field: The field name for storing the lock id [default: _id] MUST be uniquely indexed if you chage it
* token_field: The field name for storing the lock token [default: token]
* acquired_field: The field name for storing the acquisition timestamp [default: acquired_at]
* expiry_field: The field name for storing the expiry-timestamp [default: expires_at].
* database: The name of the database [required]
* collection: The name of the collection [default: lock]
*
* It is strongly recommended to put an index on the `expiry_field` for
* garbage-collection. Alternatively it's possible to automatically expire
* the locks in the database as described below:
* A TTL index MUST BE used on MongoDB 2.2+ to automatically clean up expired locks.
* Please be aware any clock drift between the application and mongo servers could
* cause locks to be released prematurely. To account for any drift;
* expireAfterSeconds can be set to a value higher than 0. The logical expiry of
* locks is handled by the application so setting a higher ``expireAfterSeconds``
* has no effect other than keeping stale data for longer.
*
* A TTL collections can be used on MongoDB 2.2+ to cleanup expired locks
* automatically. Such an index can for example look like this:
*
* db.<session-collection>.ensureIndex(
* { "<expiry-field>": 1 },
* { "expireAfterSeconds": 0 }
* db.lock.ensureIndex(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a method to createTTLIndex(). There's no need in MongoDB to create the collection, they're created on the fly by the database.

* { "expires_at": 1 },
* { "expireAfterSeconds": 60 }
* )
*
* More details on: http://docs.mongodb.org/manual/tutorial/expire-data/
* @see http://docs.mongodb.org/manual/tutorial/expire-data/
*
* Please note, the Symfony\Component\Lock\Key's $resource
* must not exceed 1024 bytes including structual overhead.
*
* @see https://docs.mongodb.com/manual/reference/limits/#Index-Key-Limit
*
* @param float $initialTtl The expiration delay of locks in seconds
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why $initialTtl is a constructor argument, whereas every other options are passed into $options?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll move it to $options.

I've been thinking it would be simpler to remove the drift option in favour of asking the developer set a slightly higher lockTTL. This would both simplify the implementation and probably make more sense to the developer rather than trying to explain why drift is a separate option. I would have to document the danger of clock drift against this Store. What are your thoughts?

*/
public function __construct(\MongoDB\Client $mongo, array $options = array(), float $initialTtl = 300.0)
public function __construct(\MongoDB\Client $mongo, array $options, float $initialTtl = 300.0)
{
if (!isset($options['database'])) {
throw new \InvalidArgumentException(
throw new InvalidArgumentException(
'You must provide the "database" option for MongoDBStore'
);
}
Expand All @@ -67,60 +71,39 @@ public function __construct(\MongoDB\Client $mongo, array $options = array(), fl

$this->options = array_merge(array(
'collection' => 'lock',
'resource_field' => '_id',
'token_field' => 'token',
'acquired_field' => 'acquired_at',
'expiry_field' => 'expires_at',
), $options);

$this->initialTtl = $initialTtl;
}

/**
* {@inheritdoc}
*
* db.lock.update(
* {
* _id: "test",
* expires_at: {
* $lte : new Date()
* }
* },
* {
* _id: "test",
* token: {# unique token #},
* acquired: new Date(),
* expires_at: new Date({# now + ttl #})
* },
* {
* upsert: 1
* }
* );
*/
public function save(Key $key)
{
$expiry = $this->createDateTime(microtime(true) + $this->initialTtl);
$now = microtime(true);
$expiry = $this->createDateTime($now + $this->initialTtl);
$token = $this->getToken($key);

$filter = array(
$this->options['resource_field'] => (string) $key,
'_id' => (string) $key,
'$or' => array(
array(
$this->options['token_field'] => $this->getToken($key),
'token' => $token,
),
array(
$this->options['expiry_field'] => array(
'$lte' => $this->createDateTime(),
'expires_at' => array(
'$lte' => $this->createDateTime($now),
),
),
),
);

$update = array(
'$set' => array(
$this->options['resource_field'] => (string) $key,
$this->options['token_field'] => $this->getToken($key),
$this->options['acquired_field'] => $this->createDateTime(),
$this->options['expiry_field'] => $expiry,
'_id' => (string) $key,
'token' => $token,
'expires_at' => $expiry,
),
);

Expand All @@ -131,8 +114,10 @@ public function save(Key $key)
$key->reduceLifetime($this->initialTtl);
try {
$this->getCollection()->updateOne($filter, $update, $options);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it atomic? on every version of mongodb? What's about réplication propagation?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MongoDB is atomic on a singular statement / document only. Hence the use of a update with filter (1 statement). MongoDB ^4.0 introduces multi statement transactions however this implementation does not rely on that. It should be atomic on all versions. Replication propagation can be controlled with a statement level options called a write concern. I'm adding support for that now. By default, mongo will read and write only to the PRIMARY node in a replica set (this is equivalent of reading and writing only to the master of a replicated RDBMS with replication slaves).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With regards to writeConcern, readConcern and readPreference I've opted to leave this to the collection fallback placing the onus on the DBA to setup the collection with alternate settings if desired. The default when left unspecified is to read / write ONLY to the PRIMARY of a replica set. This is safe for the use of locks (it's like only reading / writing to a master in an RDBMS) however an alternate strategy could be left to the DBA depending on the number of nodes they have.

I have added some documentation regarding this decision to symfony-docs

} catch (\MongoDB\Driver\Exception\BulkWriteException $e) {
} catch (\MongoDB\Driver\Exception\WriteException $e) {
throw new LockConflictedException('Failed to acquire lock', 0, $e);
} catch (\Exception $e) {
throw new LockStorageException($e->getMessage(), 0, $e);
}

if ($key->isExpired()) {
Expand All @@ -142,48 +127,32 @@ public function save(Key $key)

public function waitAndSave(Key $key)
{
throw new InvalidArgumentException(sprintf(
throw new NotSupportedException(sprintf(
'The store "%s" does not supports blocking locks.',
__CLASS__
));
}

/**
* {@inheritdoc}
*
* db.lock.update(
* {
* _id: "test",
* token: {# unique token #},
* expires_at: {
* $gte : new Date()
* }
* },
* {
* _id: "test",
* expires_at: new Date({# now + ttl #})
* },
* {
* upsert: 1
* }
* );
*/
public function putOffExpiration(Key $key, $ttl)
{
$expiry = $this->createDateTime(microtime(true) + $ttl);
$now = microtime(true);
$expiry = $this->createDateTime($now + $ttl);

$filter = array(
$this->options['resource_field'] => (string) $key,
$this->options['token_field'] => $this->getToken($key),
$this->options['expiry_field'] => array(
'$gte' => $this->createDateTime(),
'_id' => (string) $key,
'token' => $this->getToken($key),
'expires_at' => array(
'$gte' => $this->createDateTime($now),
),
);

$update = array(
'$set' => array(
$this->options['resource_field'] => (string) $key,
$this->options['expiry_field'] => $expiry,
'_id' => (string) $key,
'expires_at' => $expiry,
),
);

Expand All @@ -194,8 +163,10 @@ public function putOffExpiration(Key $key, $ttl)
$key->reduceLifetime($ttl);
try {
$this->getCollection()->updateOne($filter, $update, $options);
} catch (\MongoDB\Driver\Exception\BulkWriteException $e) {
} catch (\MongoDB\Driver\Exception\WriteException $e) {
throw new LockConflictedException('Failed to put off the expiration of the lock', 0, $e);
} catch (\Exception $e) {
throw new LockStorageException($e->getMessage(), 0, $e);
}

if ($key->isExpired()) {
Expand All @@ -208,40 +179,25 @@ public function putOffExpiration(Key $key, $ttl)

/**
* {@inheritdoc}
*
* db.lock.remove({
* _id: "test"
* });
*/
public function delete(Key $key)
{
$filter = array(
$this->options['resource_field'] => (string) $key,
$this->options['token_field'] => $this->getToken($key),
'_id' => (string) $key,
'token' => $this->getToken($key),
);

try {
$result = $this->getCollection()->deleteOne($filter);
} catch (\MongoDB\Driver\Exception\BulkWriteException $e) {
throw new LockConflictedException('Failed to delete lock', 0, $e);
}
$this->getCollection()->deleteOne($filter);
}

/**
* {@inheritdoc}
*
* db.lock.find({
* _id: "test",
* expires_at: {
* $gte : new Date()
* }
* });
*/
public function exists(Key $key)
{
$filter = array(
$this->options['resource_field'] => (string) $key,
$this->options['expiry_field'] => array(
'_id' => (string) $key,
'expires_at' => array(
'$gte' => $this->createDateTime(),
),
);
Expand Down
4 changes: 4 additions & 0 deletions src/Symfony/Component/Lock/StoreInterface.php
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
namespace Symfony\Component\Lock;

use Symfony\Component\Lock\Exception\LockConflictedException;
use Symfony\Component\Lock\Exception\LockExpiredException;
use Symfony\Component\Lock\Exception\NotSupportedException;

/**
Expand All @@ -25,6 +26,7 @@ interface StoreInterface
* Stores the resource if it's not locked by someone else.
*
* @throws LockConflictedException
* @throws LockExpiredException
*/
public function save(Key $key);

Expand All @@ -34,6 +36,7 @@ public function save(Key $key);
* If the store does not support this feature it should throw a NotSupportedException.
*
* @throws LockConflictedException
* @throws LockExpiredException
* @throws NotSupportedException
*/
public function waitAndSave(Key $key);
Expand All @@ -46,6 +49,7 @@ public function waitAndSave(Key $key);
* @param float $ttl amount of second to keep the lock in the store
*
* @throws LockConflictedException
* @throws LockExpiredException
* @throws NotSupportedException
*/
public function putOffExpiration(Key $key, $ttl);
Expand Down