INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

docs/source/howto/queryable-encryption.rst

.evergreen/run-tests.sh

django_mongodb_backend/schema.py

QE Doc flow - Three top level (or close to it) docs - How to guide on QE configuration - QE field guide (moved out of model field ref to its own section) - Topic guide on QE usage

- Planning to finish QE setup in the how-to doc then switch back to the topic doc to explain the workflow.

In the QE topics guide I'm going to explain how to develop a patient model similar to the Python QE tutorial and the encryption test suite. If embedding were possible now, I could present the same example. Since embedded models are a WIP I may either: - design the guide to work around that issue - design the guide to anticipate resolution of that issue In ether case, I'll try to wrap this 24-48 hour docs jam having covered everything I know about QE and Django and having presented it to Django folks in a way we can support.

- Configuring the encrypted_fields_map section

- Configuring the Automatic Encryption Shared Library

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

timgraham

(a very cursory review of some recent changes; I didn't comment on documentation yet)

timgraham · 2025-08-18T22:34:24Z

tests/encryption_/test_base.py

@@ -15,7 +15,7 @@

 @skipUnlessDBFeature("supports_queryable_encryption")
 @override_settings(DATABASE_ROUTERS=[TestEncryptedRouter()])
-class QueryableEncryptionTestCase(TransactionTestCase):
+class QueryableEncryptionTests(TransactionTestCase):


This is indeed a base "TestCase" as it has no tests itself. I think it's useful to keep this class for the shared skipUnless and override settings, even if setUp is removed per my comment below.

timgraham · 2025-08-18T22:36:31Z

django_mongodb_backend/fields/encryption.py

+from django.db import models
+
+
+class EncryptedFieldMixin(models.Field):


I don't remember this inheriting from Field. Was it a recent change? I don't think it should considering the field it's mixed in with should already do that inheriting.

timgraham · 2025-08-18T22:38:21Z

django_mongodb_backend/management/commands/showencryptedfieldsmap.py

+
+class Command(BaseCommand):
+    help = """
+    This command shows the mapping of encrypted fields to attributes


The old wording "Shows..." is more consistent with the built-in command's documentation. Avoid boilerplate like "This command...", "this option..."

timgraham · 2025-08-18T22:42:03Z

docs/source/ref/models/encrypted-fields.rst

+``EncryptedFieldMixin``
+=======================
+
+.. class:: EncryptedFieldMixin(models.Field)


Inheritance isn't documented here, but rather (sometimess) the __init__() signature.
Example from Django's docs: .. class:: TimestampSigner(*, key=None, sep=':', salt=None, algorithm='sha256')

timgraham · 2025-08-18T22:45:08Z

tests/encryption_/test_schema.py

+        connection = connections["encrypted"]
+        with connection.schema_editor() as editor:
+            client = connection.connection
+            encrypted_fields_map = editor._get_encrypted_fields_map(self.patient, client)


The test happens to work with either, but the first argument is supposed to be a model class rather than a model instance.

timgraham · 2025-08-18T22:47:48Z

tests/encryption_/test_base.py

+        """
+        Used in schema and field tests.
+        """


I think there's not enough shared data to justify a common setUp method. Many of these models are used in only one test, so it's wasteful to create them before every test. If it helps, you can make many of the fields nullable to make creating objects easier. (e.g. test_datefield can create a PatientRecord with only a date).

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Aug 12, 2025

View reviewed changes

docs/source/howto/queryable-encryption.rst Outdated Show resolved Hide resolved

timgraham reviewed Aug 12, 2025

View reviewed changes

docs/source/howto/queryable-encryption.rst Outdated Show resolved Hide resolved

timgraham reviewed Aug 12, 2025

View reviewed changes

docs/source/howto/queryable-encryption.rst Outdated Show resolved Hide resolved

aclark4life force-pushed the INTPYTHON-527 branch from 25b7e8b to 2469f84 Compare August 12, 2025 18:47

aclark4life commented Aug 13, 2025

View reviewed changes

.evergreen/run-tests.sh Show resolved Hide resolved

aclark4life commented Aug 13, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

aclark4life force-pushed the INTPYTHON-527 branch from cd8aa5c to 62db35a Compare August 13, 2025 14:31

aclark4life and others added 9 commits August 13, 2025 12:08

INTPYTHON-527 Add Queryable Encryption support

529db96

Code review fixes (1/x)

54f8447

QE Doc flow - Three top level (or close to it) docs - How to guide on QE configuration - QE field guide (moved out of model field ref to its own section) - Topic guide on QE usage

Code review fixes (2/2)

1294250

few edits

4c0be1b

fix code highlighting in docs

cd7c52d

Code review fixes (1/x)

08eb6c1

- Planning to finish QE setup in the how-to doc then switch back to the topic doc to explain the workflow.

Code review fixes (2/x)

7db2c94

Code review fixes (4/x)

cd035eb

- Configuring the encrypted_fields_map section

aclark4life force-pushed the INTPYTHON-527 branch from 21c237e to cd035eb Compare August 13, 2025 16:09

aclark4life added 2 commits August 13, 2025 12:47

Code review fixes (5/x)

621c44e

- Configuring the Automatic Encryption Shared Library

Code review fixes (6/6)

4fe6a35

aclark4life requested review from timgraham and Copilot August 14, 2025 00:14

This comment was marked as resolved.

Sign in to view

aclark4life requested a review from Copilot August 14, 2025 00:23

This comment was marked as resolved.

Sign in to view

Copilot review fixes

33b9b58

aclark4life force-pushed the INTPYTHON-527 branch from bff8bcf to 33b9b58 Compare August 18, 2025 19:44

aclark4life and others added 3 commits August 18, 2025 17:01

Rename/refactor tests, add stub connection test

c5bcea1

Update django_mongodb_backend/routers.py

fa428be

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update django_mongodb_backend/schema.py

3296549

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

timgraham reviewed Aug 18, 2025

View reviewed changes

		from django.db import models


		class EncryptedFieldMixin(models.Field):

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Uh oh!

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

timgraham left a comment

Choose a reason for hiding this comment

Uh oh!

timgraham Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

timgraham Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

timgraham Aug 18, 2025

aclark4life commented Jun 27, 2025 •

edited

Loading