INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

aclark4life · 2025-08-06T13:46:40Z

I'm still unclear on the user workflow. What I envisioned is:

* define model, `makemigrations`, `migrate`

* Put the output of `showencryptedfieldsmap` (which includes keyIds retrieved from server) in `AutoEncryptionOpts(encrypted_fields_map=...)`.

That is correct. We can remove --create if we never want to support this workflow:

define model
makemigrations
Put the output of showencryptedfieldsmap --create (which creates new keys) in AutoEncryptionOpts(encrypted_fields_map=...).
migrate

In other words, do we support the chicken and the egg or just the egg 😂

aclark4life · 2025-08-06T13:48:06Z

But this issue remains:

Creating test database for alias 'encrypted' ('test_djangotests-encrypted')...
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'

Any progress on this? Do we need to ask @blink1073 for help?

addaleax · 2025-08-06T14:08:22Z

@aclark4life Soooo ... I don't know what you've done around that area so far, but it's great that you're bringing it up, I didn't realize we'd have to still worry about this part because I don't think the PR currently concerns itself at all with how we integrate with crypt_shared/mongocryptd?

So I'll explain the whole story here, knowing that I'm risking telling you things you're alread well aware of 🙂

For automatic CSFLE/QE, we don't just need the libmongocrypt integration provided by the MongoDB driver, we also need a query analysis engine. This is created from part of the server source code, and comes in two forms:
1. mongocryptd, a daemon process that can be spawned and run by the driver and talked through via the MongoDB wire protocol. We consider this a legacy solution, but it is fully supported and not officially deprecated for now.
2. The mongo_crypt_v1 shared library, a dll/shared object that can be loaded by libmongocrypt at runtime.

Operationally, mongocryptd is more difficult to deploy in production applications (you're essentially managing an external daemon process from the driver, which is complex on its own, and potentially sharing this instance between multiple application processes), so while it is currently still supported, we are looking to phase it out in the long run. For backwards compatibility reasons, it is still the default (and fallback) though.

So we generally recommend using the crypt_shared library, which can be achieved by passing crypt_shared_lib_path=... to the driver, and ideally also setting crypt_shared_lib_required=True so that libmongocrypt doesn't fall back to mongocryptd and instead fails if the crypt_shared library could not be loaded.

Given that this is a new feature here, I'd strongly consider setting crypt_shared_lib_required=True and only using the crypt_shared library by default, and never using mongocryptd (or only if the user has explicitly requested it). I've asked our PMs about this in https://mongodb.slack.com/archives/C0406ECL478/p1754488650159029, it feels like the right call to me but I'd like to double-check.

As far as the tests here are concerned themselves, I imagine they're passing locally for you because you do have mongocryptd in your path, so the driver is able to spawn it. Regardless on how you decide regarding mongocryptd, you'll need to install at least one of it or the crypt_shared library in CI, and in the latter case, point the driver to the path to it (happy to provide help with the details of this should you have any questions).

aclark4life · 2025-08-06T14:26:24Z

Given that this is a new feature here, I'd strongly consider setting crypt_shared_lib_required=True and only using the crypt_shared library by default, and never using mongocryptd (or only if the user has explicitly requested it). I've asked our PMs about this in https://mongodb.slack.com/archives/C0406ECL478/p1754488650159029, it feels like the right call to me but I'd like to double-check.

Can we document our way around this by recommending crypt_shared_lib_path in production but allowing mongocryptd in development? I think that is something we can get Django folks to accept but I don't think enforcing the use of crypt shared in development would go over well.

Also what about bundling that library in the pymongocrypt wheel? It's the convenience of pip install django-mongodb-backend[encryption] that we are after and we lose that with the enterprise download step.

addaleax · 2025-08-06T14:50:59Z

Can we document our way around this by recommending crypt_shared_lib_path in production but allowing mongocryptd in development? I think that is something we can get Django folks to accept but I don't think enforcing the use of crypt shared in development would go over well.

Maybe, but I think we'd want to have a conversation around what the typical expectations here are. You'll generally want to have development and production environments behave similarly, and you'd still need to have a plan for what to do when mongocryptd does get deprecated eventually (long-term, I think it's fair to expect this to happen). You'll also still be in a position where you need to download and install mongocryptd, the only case in which this requirement goes away is the one where you happen to have the enterprise MongoDB server binaries already ready in your $PATH.

Also what about bundling that library in the pymongocrypt wheel? It's the convenience of pip install django-mongodb-backend[encryption] that we are after and we lose that with the enterprise download step.

This question comes up on a regular basis 🙂 Here's a Slack thread from April, which was one of the last times we spoke about this.

tl;dr: Yes, the setup process for CSFLE/QE is involved, and we'd like to make it easier. Currently, there is a requirement for the user to explicitly acknowledge that they have read and accepted the enterprise license agreement and that they are an Atlas or EA customer. Bundling this library with regular packages that can be installed via a regular package manager command like pip install is therefore something we can't do right now.)

Jibola

Overall looks good! I've provided comments and feedback around some documentation and clarifications, but will approve once those are addressed.

.github/workflows/runtests.py

django_mongodb_backend/model_utils.py

django_mongodb_backend/schema.py

docs/source/howto/queryable-encryption.rst

tests/encryption_/test_base.py

tests/encryption_/test_management.py

django_mongodb_backend/model_utils.py

- Fix rebase merge conflict edits - Remove integer field FIXME comments - Remove pos_int

timgraham · 2025-08-11T11:36:02Z

docs/source/topics/index.rst

@@ -11,3 +11,4 @@ know:
   embedded-models
   transactions
   known-issues
+   queryable-encryption


order above "transactions"

docs/source/ref/django-admin.rst

docs/source/ref/models/fields.rst

docs/source/howto/queryable-encryption.rst

timgraham · 2025-08-11T11:37:37Z

docs/source/ref/django-admin.rst

+    .. django-admin-option:: --create-new-keys
+
+        If specified, creates the data keys instead of getting them from the
+        database.


This option need some explanation in the howto. (use case)
Also, there is no explanation here that the command "gets the keys from the database"... need more explanation.

timgraham · 2025-08-11T12:35:18Z

docs/source/topics/queryable-encryption.rst

+.. admonition:: List of encrypted fields
+
+    See the full list of :ref:`encrypted fields <encrypted-fields>` in the
+    :doc:`Model field reference </ref/models/fields>`.


don't need to repeat the name of the linked doc "Model field reference"

timgraham · 2025-08-11T12:36:06Z

tests/encryption_/test_base.py

+
+    def setUp(self):
+        self.appointment = Appointment.objects.create(time="8:00")
+


no blank line needed after every object

timgraham · 2025-08-11T12:37:37Z

tests/encryption_/test_management.py

+        self.assertEqual(json1, json2)
+
+    def test_show_encrypted_fields_map(self):
+        self.maxDiff = None


You can set maxDiff = None at the class-level.

timgraham · 2025-08-11T12:38:50Z

docs/source/topics/queryable-encryption.rst

+.. admonition:: List of encrypted fields
+
+    See the full list of :ref:`encrypted fields <encrypted-fields>` in the
+    :doc:`Model field reference </ref/models/fields>`.


It's worth include this reference, but not in the middle of the example.

timgraham · 2025-08-11T12:39:57Z

docs/source/topics/queryable-encryption.rst

+    {
+      _id: ObjectId('68825b066fac55353a8b2b41'),
+      ssn: '123-45-6789',
+      __safeContent__: [b'\xe0)NOFB\x9a,\x08\xd7\xdd\xb8\xa6\xba$…']
+    }


I'm not sure what this is telling me. So encrypted fields appear in plain text, but what the heck is __safeContent__? As I developer, do I care about it?

Added a link to explanation.

timgraham · 2025-08-11T12:52:30Z

docs/source/topics/queryable-encryption.rst

+You can query encrypted fields using a
+:ref:`manual:qe-supported-query-operators` which must be specified in the


I'm not sure if this is the best page to link to (btw, the text renders as "using a Supported Query Operators". It doesn't show any of the options https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/encrypt-and-query/#configure-encrypted-fields-for-optimal-search-and-storage

timgraham · 2025-08-11T12:53:56Z

tests/encryption_/models.py

+
+EQUALITY_QUERY = {"queryType": "equality"}
+RANGE_QUERY = {"queryType": "range"}
+RANGE_QUERY_MIN_MAX = {"queryType": "range", "min": 0, "max": 100}


At least for the values that aren't reused, it doesn't make sense to use a module level variable.

timgraham · 2025-08-11T17:36:26Z

docs/source/howto/queryable-encryption.rst

+``AutoEncryptionOpts``.
+
+Fortunately, this is easy to do with Django MongoDB Backend. You can use
+the ``showencryptedfieldsmap`` management command to generate the schema map


:djadmin:`showencryptedfieldsmap`

timgraham · 2025-08-11T17:38:12Z

docs/source/howto/queryable-encryption.rst

+.. note:: The ``showencryptedfieldsmap`` command is only available if you
+   have the ``django_mongodb_backend`` app included in the
+   :setting:`INSTALLED_APPS` setting.


From caching docs:

.. admonition:: Didn't work? If you get the error ``Unknown command: 'createcachecollection'``, ensure ``"django_mongodb_backend"`` is in your :setting:`INSTALLED_APPS` setting.

timgraham · 2025-08-11T17:38:58Z

docs/source/howto/queryable-encryption.rst

+To generate the encrypted fields map, run the following command in your Django
+project::
+
+    python manage.py showencryptedfieldsmap


If following the above steps, --database=encrypted is also needed.

timgraham · 2025-08-11T17:56:56Z

tests/encryption_/test_base.py

+            ]
+        }
+        connection = connections["encrypted"]
+        auto_encryption_opts = getattr(connection.connection._options, "auto_encryption_opts", None)


getattr should be unnecessary. This test shouldn't run if auto_encryption_opt isn't set.

tests/encryption_/routers.py

timgraham · 2025-08-11T18:02:32Z

docs/source/howto/queryable-encryption.rst

+
+In addition to :doc:`installing </intro/install>` and
+:doc:`configuring </intro/configure>` Django MongoDB Backend,
+you will need to install PyMongo with Queryable Encryption support::


I think saying "you will need to install PyMongo with Queryable Encryption support" and then giving a command that doesn't mention pymongo could be confusing.

Probably "you need to install some additional dependencies::" could be enough.

timgraham · 2025-08-11T18:04:15Z

django_mongodb_backend/schema.py

+        else:
+            db.create_collection(db_table)
+
+    def _get_encrypted_fields_map(self, model, client, auto_encryption_opts, from_db=False):


from_db is not very descriptive. How about create_data_keys?

timgraham · 2025-08-11T18:12:07Z

django_mongodb_backend/base.py

+        # Avoid using PyMongo to check the database version or require
+        # pymongocrypt>=1.14.2 which will contain a fix for the `buildInfo`
+        # command. https://jira.mongodb.org/browse/PYTHON-5429
+        return tuple(self.connection.admin.command("buildInfo")["versionArray"])


This still uses PyMongo. I guess you meant the server_info() method. Is there a big benefit to switching back to in the the future? If so, you can write something like "This can use tuple(self.connection.server_info()["versionArray"]) on pymongocrypt 1.14.2+" (heck, if there's a benefit, you can add some conditional code based on the pymongocrypt version)... pymongocrypt 1.15.0 was released Aug. 15 (1.14.1 is the latest 1.14.x release so unclear if there will be a 1.14.2). Anyway, some comment update is needed.

timgraham · 2025-08-11T18:48:28Z

django_mongodb_backend/management/commands/showencryptedfieldsmap.py

+
+    def handle(self, *args, **options):
+        db = options["database"]
+        create = options.get("create", False)


Suggested change

create = options.get("create", False)

create = options.get("`create_new_keys`", False)

timgraham · 2025-08-11T18:49:33Z

django_mongodb_backend/management/commands/showencryptedfieldsmap.py

+            for model in router.get_migratable_models(app_config, db):
+                if model_has_encrypted_fields(model):
+                    from_db = not create
+                    fields = connection.schema_editor()._get_encrypted_fields_map(


use with connection.schema_editor() as editor: at the top-level outside of the loops so you don't have to initialize a new one for every model.

timgraham · 2025-08-11T18:50:28Z

django_mongodb_backend/schema.py

+        else:
+            db.create_collection(db_table)
+
+    def _get_encrypted_fields_map(self, model, client, auto_encryption_opts, from_db=False):


auto_encryption_opts needn't be an argument. It's an attribute of client.

timgraham · 2025-08-11T18:51:31Z

django_mongodb_backend/schema.py

+                raise ImproperlyConfigured(
+                    "Encrypted fields found but "
+                    "DATABASES[[self.connection.alias}]['OPTIONS'] is missing "
+                    "auto_encryption_opts. Please set `auto_encryption_opts` "
+                    "in the connection settings."
+                )


Did you try adding a test for this exception?

I've tested it manually but haven't written a test for it yet.

timgraham · 2025-08-11T18:54:55Z

django_mongodb_backend/schema.py

+            else:
+                # If the encrypted fields map is provided, get the map for the
+                # specific collection.
+                encrypted_fields_map = encrypted_fields_map.get(db_table, None)


What does it mean to fallback to None?

timgraham · 2025-08-11T19:02:09Z

docs/source/howto/queryable-encryption.rst

+        "KMS_PROVIDERS": {},
+        "KMS_CREDENTIALS": {},


For KMS_CREDENTIALS, I believe the options are documented here: https://pymongo.readthedocs.io/en/stable/api/pymongo/encryption.html#pymongo.encryption.ClientEncryption.create_data_key (but pymongo docs are deprecated?)

So a minimal example for AWS: "KMS_CREDENTIALS": {"aws": {"region": "...", "key": "..."}

Don't break the build!

Don't break the build Part II

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life requested review from timgraham, addaleax, Jibola, WaVEV and Copilot August 5, 2025 22:19

This comment was marked as resolved.

Sign in to view

Jibola requested changes Aug 6, 2025

View reviewed changes

aclark4life force-pushed the INTPYTHON-527 branch from ae765eb to 886b1e0 Compare August 8, 2025 20:50

aclark4life commented Aug 8, 2025

View reviewed changes

django_mongodb_backend/model_utils.py Show resolved Hide resolved

INTPYTHON-527 Add Queryable Encryption support

4496054

aclark4life force-pushed the INTPYTHON-527 branch from 886b1e0 to 4496054 Compare August 11, 2025 10:40

Code review fixes

bddae41

- Fix rebase merge conflict edits - Remove integer field FIXME comments - Remove pos_int

aclark4life force-pushed the INTPYTHON-527 branch from 5b2bc07 to bddae41 Compare August 11, 2025 11:09

Update release notes & related docs

bf1e6d7

timgraham reviewed Aug 11, 2025

View reviewed changes

aclark4life added 5 commits August 11, 2025 15:27

Code review fixes (1/x)

a566237

Code review fixes (2/x)

d6ab11e

Code review fixes (3/x)

ecc386d

Don't break the build!

Code review fixes (4/x)

20de1f2

Don't break the build Part II

Code review fixes (5/x)

2d4b8b7


		def setUp(self):
		self.appointment = Appointment.objects.create(time="8:00")

		You can query encrypted fields using a
		:ref:`manual:qe-supported-query-operators` which must be specified in the

	create = options.get("create", False)
	create = options.get("`create_new_keys`", False)

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

aclark4life commented Aug 6, 2025

Uh oh!

aclark4life commented Aug 6, 2025

Uh oh!

addaleax commented Aug 6, 2025

Uh oh!

aclark4life commented Aug 6, 2025

Uh oh!

addaleax commented Aug 6, 2025

Uh oh!

Jibola left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025 •

edited

Loading

timgraham Aug 11, 2025 •

edited

Loading