INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/test_fields.py

timgraham · 2025-08-12T10:43:43Z

tests/encryption_/test_base.py

+    def setUp(self):
+        self.appointment = Appointment.objects.create(time="8:00")
+        self.billing = Billing.objects.create(
+            cc_type="Visa", cc_number=1234567890123456, account_balance=100.50
+        )
+        self.portal_user = PatientPortalUser.objects.create(
+            ip_address="127.0.0.1",
+            url="https://example.com",
+        )
+        self.patientrecord = PatientRecord.objects.create(
+            ssn="123-45-6789",
+            birth_date="1970-01-01",
+            profile_picture=b"image data",
+            weight=175.5,
+            patient_age=47,
+        )
+        self.patient = Patient.objects.create(
+            patient_id=1,
+            patient_name="John Doe",
+            patient_notes="patient notes " * 25,
+            registration_date=datetime(2023, 10, 1, 12, 0, 0),
+            is_active=True,
+            email="john.doe@example.com",
+        )
+
+        # TODO: Embed billing and patient_record models in patient model
+        # then add tests


I think the test data is only needed for the fields tests? It looks like most test data is only used in one test method, so it's a bit wasteful to create it before every test method.

It's used in schema and fields

timgraham · 2025-08-12T10:48:55Z

tests/encryption_/test_fields.py

+        """
+        Fields that have not been tested elsewhere.
+        """


It's difficult to navigate to navigate this test file and confirm that all fields are tested. Rather than a test method for each method, it would be more clear to have a test method for each field, e.g. test_charfield.

Just to be clear, don't move the misc untested fields to the patient themed tests right ? That's what I'm inclined to do but I believe you are suggesting the opposite. I like the patient themed tests but we could also re-do as field-first focused.

It's fine to keep the models as is. I'm suggesting that each field have its own test method.

tests/encryption_/routers.py

tests/encryption_/models.py

timgraham · 2025-08-12T11:27:16Z

tests/encryption_/test_management.py

+    def test_create_new_keys(self):
+        out = StringIO()
+        call_command(
+            "showencryptedfieldsmap",
+            "--database",
+            "encrypted",
+            "--create-new-keys",
+            verbosity=0,
+            stdout=out,
+        )
+        command_output = json_util.loads(out.getvalue())
+
+        # TODO: Until the todo below is fixed, you can test this feature
+        # manually by uncommenting the following line and pasting the output
+        # into your client-side `encrypted_fields_map` configuration. You also
+        # need to temporarily configure server-side encryption for this to
+        # procedure to work.
+        # print(command_output)
+        self._compare_output(
+            self.expected_patient_record,
+            command_output["encryption__patientrecord"],
+        )
+
+        # Create a new connection to verify that the keys can be used in a
+        # client-side configuration to migrate the encrypted fields.
+        conn_params = connections["encrypted"].get_connection_params()
+        auto_encryption_opts = AutoEncryptionOpts(
+            key_vault_namespace="encryption.__keyvault",
+            kms_providers={"local": {"key": os.urandom(96)}},
+            encrypted_fields_map=command_output,
+        )
+        if conn_params.pop("auto_encryption_opts", False):
+            # Call MongoClient instead of get_new_connection because
+            # get_new_connection will return the encrypted connection from the
+            # connection pool.
+            with pymongo.MongoClient(**conn_params, auto_encryption_opts=auto_encryption_opts):
+                call_command("migrate", "--database", "encrypted", verbosity=0)
+
+        # TODO: Check the key vault to ensure that the keys created by
+        # `showencryptedfieldsmap --create-new-keys` are in the key vault.


There is more to do to fix this test than just add assertions. The migrate step does nothing because there are no pending migrations to apply the encrypted database. Testing this is complicated and I'm not sure this attempted path is viable.

django_mongodb_backend/schema.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

docs/source/ref/queryable-encryption.rst

timgraham · 2025-08-12T15:29:01Z

docs/source/ref/django-admin.rst

+        If specified, this option will create new encryption keys for use in
+        production workflows.


"for use in production workflows" need some explanation

timgraham · 2025-08-12T15:32:48Z

docs/source/ref/settings.rst

+Queryable Encryption requires a KMS provider to encrypt and decrypt data. Django
+MongoDB Backend supports configuring KMS credentials and providers for Queryable
+Encryption via the ``KMS_CREDENTIALS`` setting in the ``DATABASES``
+configuration and the ``kms_provider`` method on the ``DatabaseRouter``.
+
+E.g. to configure AWS KMS credentials:
+
+.. code-block:: python
+
+    KMS_CREDENTIALS = {
+        "aws": {
+            "key": os.getenv("AWS_KEY_ARN", ""),
+            "region": os.getenv("AWS_KEY_REGION", ""),
+        },
+    }
+    DATABASES = {
+        # …
+    }
+    DATABASES["encrypted"]["KMS_CREDENTIALS"] = KMS_CREDENTIALS


I imagined this as a section at the bottom of the howto "Configuring KMS Providers", instead of the "KMS providers and credentials" admonition box that links (not really helpfully) to MongoDB's docs. It will also discuss router.kms_provider.

timgraham · 2025-08-12T15:36:59Z

docs/source/howto/queryable-encryption.rst

+Encrypted fields map
+~~~~~~~~~~~~~~~~~~~~
+
+In addition to the :ref:`settings described in the how-to guide
+<queryable-encryption-settings>` you will need to provide a
+``encrypted_fields_map`` to the ``AutoEncryptionOpts``.
+
+You can use the :djadmin:`showencryptedfieldsmap` management command to generate
+the schema map for your encrypted fields and then use the results in your
+settings::
+
+    python manage.py showencryptedfieldsmap --database=encrypted


We're missing the step on defining your models (consolidate from topic guide?), then makemigrations and migrate. Otherwise showencryptedfieldsmap may crash without any keyIds to retrieve?

timgraham · 2025-08-12T15:39:20Z

docs/source/howto/queryable-encryption.rst

+In addition to the :ref:`settings described in the how-to guide
+<queryable-encryption-settings>` you will need to provide a
+``encrypted_fields_map`` to the ``AutoEncryptionOpts``.


But this step is optional for development, right? The howto doesn't explain how --create-new-keys fits in.

timgraham · 2025-08-12T15:55:43Z

docs/source/howto/queryable-encryption.rst

        "OPTIONS": {
            "auto_encryption_opts": AutoEncryptionOpts(
-                …
+                ...


@aclark4life fyi: use three dots rather than the ellipses character. The entire code block won't be highlighted if the syntax is invalid, and the ellipsis character isn't valid.

aclark4life · 2025-08-13T00:15:25Z

.evergreen/run-tests.sh

@@ -6,6 +6,7 @@ set -eux
 /opt/python/3.10/bin/python3 -m venv venv
 . venv/bin/activate
 python -m pip install -U pip
+pip install ".[encryption]"


Any objection to combining this line and the next line into pip install -e .[encryption]?

It would be best to have this line run only on builds that test encryption, so that we don't inadvertently add some top-level import for an optional dependency (similar reason to why we have a separate build for GIS that installs gdal-bin.) Perhaps the encryption build could set an environment variable like RUNNING_ENCRYPTION_TESTS that this script could consult.

aclark4life · 2025-08-13T00:18:17Z

django_mongodb_backend/schema.py

 from .fields import EmbeddedModelField
 from .gis.schema import GISSchemaEditor
+from .indexes import SearchIndex


@timgraham Not sure if you noticed this yet but in anticipation should I revert and leave for a sep PR ?

That would be cleanest. Perhaps there are other offenders, though sometimes importing the full path might be preferable to a large number of dots. Some is some debate on this: django/django#19707, django/django#18813

QE Doc flow - Three top level (or close to it) docs - How to guide on QE configuration - QE field guide (moved out of model field ref to its own section) - Topic guide on QE usage

- Planning to finish QE setup in the how-to doc then switch back to the topic doc to explain the workflow.

In the QE topics guide I'm going to explain how to develop a patient model similar to the Python QE tutorial and the encryption test suite. If embedding were possible now, I could present the same example. Since embedded models are a WIP I may either: - design the guide to work around that issue - design the guide to anticipate resolution of that issue In ether case, I'll try to wrap this 24-48 hour docs jam having covered everything I know about QE and Django and having presented it to Django folks in a way we can support.

- Configuring the encrypted_fields_map section

- Configuring the Automatic Encryption Shared Library

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life requested review from timgraham, addaleax, Copilot and Jibola August 12, 2025 03:31

This comment was marked as resolved.

Sign in to view

mongodb deleted a comment from Copilot AI Aug 12, 2025

aclark4life force-pushed the INTPYTHON-527 branch from 0997357 to 160f7a9 Compare August 12, 2025 10:46

timgraham reviewed Aug 12, 2025

View reviewed changes

docs/source/ref/queryable-encryption.rst Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Aug 12, 2025

View reviewed changes

aclark4life force-pushed the INTPYTHON-527 branch from 25b7e8b to 2469f84 Compare August 12, 2025 18:47

aclark4life commented Aug 13, 2025

View reviewed changes

aclark4life force-pushed the INTPYTHON-527 branch from cd8aa5c to 62db35a Compare August 13, 2025 14:31

aclark4life and others added 9 commits August 13, 2025 12:08

INTPYTHON-527 Add Queryable Encryption support

529db96

Code review fixes (1/x)

54f8447

QE Doc flow - Three top level (or close to it) docs - How to guide on QE configuration - QE field guide (moved out of model field ref to its own section) - Topic guide on QE usage

Code review fixes (2/2)

1294250

few edits

4c0be1b

fix code highlighting in docs

cd7c52d

Code review fixes (1/x)

08eb6c1

- Planning to finish QE setup in the how-to doc then switch back to the topic doc to explain the workflow.

Code review fixes (2/x)

7db2c94

Code review fixes (4/x)

cd035eb

- Configuring the encrypted_fields_map section

aclark4life force-pushed the INTPYTHON-527 branch from 21c237e to cd035eb Compare August 13, 2025 16:09

Code review fixes (5/x)

621c44e

- Configuring the Automatic Encryption Shared Library

		If specified, this option will create new encryption keys for use in
		production workflows.

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025 •

edited

Loading