-
Notifications
You must be signed in to change notification settings - Fork 123
Description
Hi @benalexau - you've been so helpful... I was wondering if you could sanity check the below... if not, or if there's a better forum for these types of questions please let me know!
LMDB really does seem to be best-in-class for read focused performance - I'm surprised there's not more people using/discussing it... it's made it challenging to get as far as I've got!
Background: I've been using LMDB for a while with a lot of success - all my keys are incrementing longs (put with MDB_APPEND flag) and my values are binary encoded, compressed (using LZ4-fast) with checksums (using xxHash64).
My questions;
-
I've been using LMDB a bit like a relational database and creating a separate LMDB file/Env for each entity (i.e. think SQL table). This works very well, but there are rare cases where I have to open transaction open across multiple LMDB file/Env's.
-
I group things like foreign-key relationships into the same LMDB file/Env as a separate named database to avoid having to always open multiple read or write transactions.
-
There are cases where I want to partition information - so I have a separate LMDB file/Env for each calendar month or complex user specific databases - so I keep only the most commonly/recently used LMDB file/Env's open to limit the number of environments I have open at a time.
-
There are rare (and performance critical) cases where I want to upgrade from a read lock to a write lock. In these cases, I first release the read lock, then open the write lock and re-read everything again with this write lock. I've noted from What does Environment maxreaders reached (-30790) actually mean? #65 (comment) that you prefer JVM-based locking. I thought I might get performance gains by using JVM-based locking and setting EnvFlags.MDB_NOLOCK - but didn't... so gave up.
Is there anything obviously wrong with any of the above? Any recommendations?
Thanks in advance ;-)