Skip to content

Q: General questions / COVID-19 sanity checking #150

@ghudo

Description

@ghudo

Hi @benalexau - you've been so helpful... I was wondering if you could sanity check the below... if not, or if there's a better forum for these types of questions please let me know!

LMDB really does seem to be best-in-class for read focused performance - I'm surprised there's not more people using/discussing it... it's made it challenging to get as far as I've got!

Background: I've been using LMDB for a while with a lot of success - all my keys are incrementing longs (put with MDB_APPEND flag) and my values are binary encoded, compressed (using LZ4-fast) with checksums (using xxHash64).

My questions;

  1. I've been using LMDB a bit like a relational database and creating a separate LMDB file/Env for each entity (i.e. think SQL table). This works very well, but there are rare cases where I have to open transaction open across multiple LMDB file/Env's.

  2. I group things like foreign-key relationships into the same LMDB file/Env as a separate named database to avoid having to always open multiple read or write transactions.

  3. There are cases where I want to partition information - so I have a separate LMDB file/Env for each calendar month or complex user specific databases - so I keep only the most commonly/recently used LMDB file/Env's open to limit the number of environments I have open at a time.

  4. There are rare (and performance critical) cases where I want to upgrade from a read lock to a write lock. In these cases, I first release the read lock, then open the write lock and re-read everything again with this write lock. I've noted from What does Environment maxreaders reached (-30790) actually mean? #65 (comment) that you prefer JVM-based locking. I thought I might get performance gains by using JVM-based locking and setting EnvFlags.MDB_NOLOCK - but didn't... so gave up.

Is there anything obviously wrong with any of the above? Any recommendations?

Thanks in advance ;-)

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionQueries on usage of the library, architectural patterns and alike

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions