INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

timgraham · 2025-08-02T23:39:06Z

The encryption tests are passing locally for me on Enterprise and on the Atlas VM.

On GitHub actions, this first issue was solved by adding "directConnection": True in DATABASES:

  File "/home/runner/work/django-mongodb-backend/django-mongodb-backend/django_repo/django/db/backends/base/base.py", line 197, in check_database_version_supported
    and self.get_database_version() < self.features.minimum_database_version
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-mongodb-backend/django-mongodb-backend/django_mongodb_backend/base.py", line 235, in get_database_version
    return tuple(self.connection.admin.command("buildInfo")["versionArray"])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 926, in command
    with self._client._conn_for_reads(read_preference, session, operation=command_name) as (
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1864, in _conn_for_reads
    server = self._select_server(read_preference, session, operation)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1812, in _select_server
    server = topology.select_server(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 409, in select_server
    server = self._select_server(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 387, in _select_server
    servers = self.select_servers(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 294, in select_servers
    server_descriptions = self._select_servers_loop(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 344, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: Could not reach any servers in [('3ff0eef351ff', 27017)]. Replica set is configured with internal hostnames or IPs?, Timeout: 30s, Topology Description: <TopologyDescription id: 688e9488ca8c88d98365da45, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('3ff0eef351ff', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('3ff0eef351ff:27017: [Errno -3] Temporary failure in name resolution (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>

But this issue remains:

Creating test database for alias 'encrypted' ('test_djangotests-encrypted')...
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
  Applying sites.0002_alter_domain_unique... OK
Operations to perform:
  Synchronize unmigrated apps: auth, contenttypes, encryption_, messages, sessions, staticfiles
  Apply all migrations: admin, sites
Synchronizing apps without migrations:
  Creating tables...
    Creating table encryption__appointment
Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/encryption.py", line 286, in mark_command
    res = self.mongocryptd_client[database].command(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 926, in command
    with self._client._conn_for_reads(read_preference, session, operation=command_name) as (
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1864, in _conn_for_reads
    server = self._select_server(read_preference, session, operation)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1812, in _select_server
    server = topology.select_server(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 409, in select_server
    server = self._select_server(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 387, in _select_server
    servers = self.select_servers(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 294, in select_servers
    server_descriptions = self._select_servers_loop(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 344, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27020: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 10.0s, Topology Description: <TopologyDescription id: 688e9f76697ba6965a378048, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27020) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27020: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>

tests/encryption_/tests.py

docs/source/faq.rst

tests/encryption_/tests.py

docs/source/howto/queryable-encryption.rst

docs/source/topics/known-issues.rst

docs/source/topics/queryable-encryption.rst

tests/encryption_/tests.py

django_mongodb_backend/management/commands/showfieldsmap.py

django_mongodb_backend/schema.py

docs/source/faq.rst

tests/encryption_/models.py

docs/source/faq.rst

django_mongodb_backend/fields/__init__.py

django_mongodb_backend/schema.py

docs/source/howto/queryable-encryption.rst

django_mongodb_backend/management/commands/showfieldsmap.py

aclark4life · 2025-08-06T13:46:40Z

I'm still unclear on the user workflow. What I envisioned is:

* define model, `makemigrations`, `migrate`

* Put the output of `showencryptedfieldsmap` (which includes keyIds retrieved from server) in `AutoEncryptionOpts(encrypted_fields_map=...)`.

That is correct. We can remove --create if we never want to support this workflow:

define model
makemigrations
Put the output of showencryptedfieldsmap --create (which creates new keys) in AutoEncryptionOpts(encrypted_fields_map=...).
migrate

In other words, do we support the chicken and the egg or just the egg 😂

aclark4life · 2025-08-06T13:48:06Z

But this issue remains:

Creating test database for alias 'encrypted' ('test_djangotests-encrypted')...
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'

Any progress on this? Do we need to ask @blink1073 for help?

addaleax · 2025-08-06T14:08:22Z

@aclark4life Soooo ... I don't know what you've done around that area so far, but it's great that you're bringing it up, I didn't realize we'd have to still worry about this part because I don't think the PR currently concerns itself at all with how we integrate with crypt_shared/mongocryptd?

So I'll explain the whole story here, knowing that I'm risking telling you things you're alread well aware of 🙂

For automatic CSFLE/QE, we don't just need the libmongocrypt integration provided by the MongoDB driver, we also need a query analysis engine. This is created from part of the server source code, and comes in two forms:
1. mongocryptd, a daemon process that can be spawned and run by the driver and talked through via the MongoDB wire protocol. We consider this a legacy solution, but it is fully supported and not officially deprecated for now.
2. The mongo_crypt_v1 shared library, a dll/shared object that can be loaded by libmongocrypt at runtime.

Operationally, mongocryptd is more difficult to deploy in production applications (you're essentially managing an external daemon process from the driver, which is complex on its own, and potentially sharing this instance between multiple application processes), so while it is currently still supported, we are looking to phase it out in the long run. For backwards compatibility reasons, it is still the default (and fallback) though.

So we generally recommend using the crypt_shared library, which can be achieved by passing crypt_shared_lib_path=... to the driver, and ideally also setting crypt_shared_lib_required=True so that libmongocrypt doesn't fall back to mongocryptd and instead fails if the crypt_shared library could not be loaded.

Given that this is a new feature here, I'd strongly consider setting crypt_shared_lib_required=True and only using the crypt_shared library by default, and never using mongocryptd (or only if the user has explicitly requested it). I've asked our PMs about this in https://mongodb.slack.com/archives/C0406ECL478/p1754488650159029, it feels like the right call to me but I'd like to double-check.

As far as the tests here are concerned themselves, I imagine they're passing locally for you because you do have mongocryptd in your path, so the driver is able to spawn it. Regardless on how you decide regarding mongocryptd, you'll need to install at least one of it or the crypt_shared library in CI, and in the latter case, point the driver to the path to it (happy to provide help with the details of this should you have any questions).

aclark4life · 2025-08-06T14:26:24Z

Given that this is a new feature here, I'd strongly consider setting crypt_shared_lib_required=True and only using the crypt_shared library by default, and never using mongocryptd (or only if the user has explicitly requested it). I've asked our PMs about this in https://mongodb.slack.com/archives/C0406ECL478/p1754488650159029, it feels like the right call to me but I'd like to double-check.

Can we document our way around this by recommending crypt_shared_lib_path in production but allowing mongocryptd in development? I think that is something we can get Django folks to accept but I don't think enforcing the use of crypt shared in development would go over well.

Also what about bundling that library in the pymongocrypt wheel? It's the convenience of pip install django-mongodb-backend[encryption] that we are after and we lose that with the enterprise download step.

addaleax · 2025-08-06T14:50:59Z

Can we document our way around this by recommending crypt_shared_lib_path in production but allowing mongocryptd in development? I think that is something we can get Django folks to accept but I don't think enforcing the use of crypt shared in development would go over well.

Maybe, but I think we'd want to have a conversation around what the typical expectations here are. You'll generally want to have development and production environments behave similarly, and you'd still need to have a plan for what to do when mongocryptd does get deprecated eventually (long-term, I think it's fair to expect this to happen). You'll also still be in a position where you need to download and install mongocryptd, the only case in which this requirement goes away is the one where you happen to have the enterprise MongoDB server binaries already ready in your $PATH.

Also what about bundling that library in the pymongocrypt wheel? It's the convenience of pip install django-mongodb-backend[encryption] that we are after and we lose that with the enterprise download step.

This question comes up on a regular basis 🙂 Here's a Slack thread from April, which was one of the last times we spoke about this.

tl;dr: Yes, the setup process for CSFLE/QE is involved, and we'd like to make it easier. Currently, there is a requirement for the user to explicitly acknowledge that they have read and accepted the enterprise license agreement and that they are an Atlas or EA customer. Bundling this library with regular packages that can be installed via a regular package manager command like pip install is therefore something we can't do right now.)

Jibola

Overall looks good! I've provided comments and feedback around some documentation and clarifications, but will approve once those are addressed.

.github/workflows/runtests.py

django_mongodb_backend/model_utils.py

django_mongodb_backend/schema.py

docs/source/howto/queryable-encryption.rst

tests/encryption_/test_base.py

tests/encryption_/test_management.py

django_mongodb_backend/model_utils.py

- Fix rebase merge conflict edits - Remove integer field FIXME comments - Remove pos_int

timgraham · 2025-08-11T11:36:02Z

docs/source/topics/index.rst

@@ -11,3 +11,4 @@ know:
   embedded-models
   transactions
   known-issues
+   queryable-encryption


order above "transactions"

timgraham · 2025-08-11T11:36:20Z

docs/source/ref/django-admin.rst

+
+``showencryptedfieldsmap``
+--------------------------
+


.. versionadded:: 5.2.0b2

timgraham · 2025-08-11T11:36:25Z

docs/source/ref/models/fields.rst

+
+Encrypted fields
+----------------
+


.. versionadded:: 5.2.0b2

timgraham · 2025-08-11T11:36:32Z

docs/source/howto/queryable-encryption.rst

+================================
+Configuring Queryable Encryption
+================================
+


.. versionadded:: 5.2.0b2

timgraham · 2025-08-11T11:37:37Z

docs/source/ref/django-admin.rst

+    .. django-admin-option:: --create-new-keys
+
+        If specified, creates the data keys instead of getting them from the
+        database.


This option need some explanation in the howto. (use case)
Also, there is no explanation here that the command "gets the keys from the database"... need more explanation.

timgraham · 2025-08-11T12:35:18Z

docs/source/topics/queryable-encryption.rst

+.. admonition:: List of encrypted fields
+
+    See the full list of :ref:`encrypted fields <encrypted-fields>` in the
+    :doc:`Model field reference </ref/models/fields>`.


don't need to repeat the name of the linked doc "Model field reference"

timgraham · 2025-08-11T12:36:06Z

tests/encryption_/test_base.py

+
+    def setUp(self):
+        self.appointment = Appointment.objects.create(time="8:00")
+


no blank line needed after every object

timgraham · 2025-08-11T12:37:37Z

tests/encryption_/test_management.py

+        self.assertEqual(json1, json2)
+
+    def test_show_encrypted_fields_map(self):
+        self.maxDiff = None


You can set maxDiff = None at the class-level.

timgraham · 2025-08-11T12:38:50Z

docs/source/topics/queryable-encryption.rst

+.. admonition:: List of encrypted fields
+
+    See the full list of :ref:`encrypted fields <encrypted-fields>` in the
+    :doc:`Model field reference </ref/models/fields>`.


It's worth include this reference, but not in the middle of the example.

timgraham · 2025-08-11T12:39:57Z

docs/source/topics/queryable-encryption.rst

+    {
+      _id: ObjectId('68825b066fac55353a8b2b41'),
+      ssn: '123-45-6789',
+      __safeContent__: [b'\xe0)NOFB\x9a,\x08\xd7\xdd\xb8\xa6\xba$…']
+    }


I'm not sure what this is telling me. So encrypted fields appear in plain text, but what the heck is "safeContent"? As I developer, do I care about it?

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Aug 2, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life force-pushed the INTPYTHON-527 branch from 6ab0a86 to fb1e120 Compare August 3, 2025 18:37

addaleax reviewed Aug 4, 2025

View reviewed changes

docs/source/faq.rst Outdated Show resolved Hide resolved

timgraham reviewed Aug 4, 2025

View reviewed changes

timgraham reviewed Aug 5, 2025

View reviewed changes

django_mongodb_backend/management/commands/showfieldsmap.py Outdated Show resolved Hide resolved

aclark4life requested review from timgraham, addaleax, Jibola, WaVEV and Copilot August 5, 2025 22:19

This comment was marked as resolved.

Sign in to view

Jibola requested changes Aug 6, 2025

View reviewed changes

aclark4life force-pushed the INTPYTHON-527 branch from ae765eb to 886b1e0 Compare August 8, 2025 20:50

aclark4life commented Aug 8, 2025

View reviewed changes

django_mongodb_backend/model_utils.py Show resolved Hide resolved

INTPYTHON-527 Add Queryable Encryption support

4496054

aclark4life force-pushed the INTPYTHON-527 branch from 886b1e0 to 4496054 Compare August 11, 2025 10:40

Code review fixes

bddae41

- Fix rebase merge conflict edits - Remove integer field FIXME comments - Remove pos_int

aclark4life force-pushed the INTPYTHON-527 branch from 5b2bc07 to bddae41 Compare August 11, 2025 11:09

Update release notes & related docs

bf1e6d7

timgraham reviewed Aug 11, 2025

View reviewed changes


		def setUp(self):
		self.appointment = Appointment.objects.create(time="8:00")


		``showencryptedfieldsmap``
		--------------------------


		Encrypted fields
		----------------

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

timgraham commented Aug 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025 •

edited

Loading