INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

docs/source/ref/django-admin.rst

docs/source/ref/settings.rst

timgraham · 2025-08-12T15:36:59Z

docs/source/howto/queryable-encryption.rst

+Encrypted fields map
+~~~~~~~~~~~~~~~~~~~~
+
+In addition to the :ref:`settings described in the how-to guide
+<queryable-encryption-settings>` you will need to provide a
+``encrypted_fields_map`` to the ``AutoEncryptionOpts``.
+
+You can use the :djadmin:`showencryptedfieldsmap` management command to generate
+the schema map for your encrypted fields and then use the results in your
+settings::
+
+    python manage.py showencryptedfieldsmap --database=encrypted


We're missing the step on defining your models (consolidate from topic guide?), then makemigrations and migrate. Otherwise showencryptedfieldsmap may crash without any keyIds to retrieve?

fixed and no, it should return "no keys found"

docs/source/howto/queryable-encryption.rst

.evergreen/run-tests.sh

django_mongodb_backend/schema.py

QE Doc flow - Three top level (or close to it) docs - How to guide on QE configuration - QE field guide (moved out of model field ref to its own section) - Topic guide on QE usage

- Planning to finish QE setup in the how-to doc then switch back to the topic doc to explain the workflow.

In the QE topics guide I'm going to explain how to develop a patient model similar to the Python QE tutorial and the encryption test suite. If embedding were possible now, I could present the same example. Since embedded models are a WIP I may either: - design the guide to work around that issue - design the guide to anticipate resolution of that issue In ether case, I'll try to wrap this 24-48 hour docs jam having covered everything I know about QE and Django and having presented it to Django folks in a way we can support.

- Configuring the encrypted_fields_map section

- Configuring the Automatic Encryption Shared Library

…map.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull Request Overview

This PR implements comprehensive Queryable Encryption support for the Django MongoDB Backend, enabling developers to use encrypted fields in Django models while maintaining the ability to query on encrypted data.

Key changes:

Added encrypted field types that extend Django's standard fields with encryption capabilities
Implemented automatic encrypted collection creation with proper schema generation
Added management command for viewing and managing encrypted field mappings

Reviewed Changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`django_mongodb_backend/fields/encryption.py`	Implements `EncryptedFieldMixin` and all encrypted field variants
`django_mongodb_backend/schema.py`	Adds encrypted collection creation and encrypted fields map generation
`django_mongodb_backend/management/commands/showencryptedfieldsmap.py`	Management command for displaying encrypted field mappings
`django_mongodb_backend/features.py`	Adds feature detection for Queryable Encryption support
`tests/encryption_/`	Comprehensive test suite for encrypted fields and functionality
`docs/source/`	Complete documentation for setup and usage

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-14T00:24:28Z

django_mongodb_backend/schema.py

+        options = client._options
+        auto_encryption_opts = options.auto_encryption_opts
+        kms_provider = router.kms_provider(model)
+        master_key = self.connection.settings_dict.get("KMS_CREDENTIALS").get(kms_provider)


This code will raise an AttributeError if 'KMS_CREDENTIALS' is not in settings_dict, as .get() returns None and then .get() is called on None. Should use: master_key = self.connection.settings_dict.get("KMS_CREDENTIALS", {}).get(kms_provider)

Suggested change

master_key = self.connection.settings_dict.get("KMS_CREDENTIALS").get(kms_provider)

master_key = self.connection.settings_dict.get("KMS_CREDENTIALS", {}).get(kms_provider)

Copilot · 2025-08-14T00:24:28Z

django_mongodb_backend/routers.py

        if not model_name:
            return None
        try:
            model = apps.get_model(app_label, model_name)
        except LookupError:
            return None
        return False if issubclass(model, EmbeddedModel) else None
+
+


This function is defined at module level but references 'self' as if it's a method. It should either be a standalone function without 'self' parameter or be properly defined as a method within a class.

Suggested change

# This function is intended to be monkey-patched as a method of ConnectionRouter.

Copilot · 2025-08-14T00:24:28Z

tests/encryption_/test_management.py

+    def _compare_output(self, json1, json2):
+        # Remove keyIds since they are different for each run.
+        for field in json2["fields"]:
+            del field["keyId"]


Using del on a dictionary key that might not exist will raise a KeyError. Should use field.pop("keyId", None) to safely remove the key if it exists, similar to line 62 in test_schema.py.

Suggested change

del field["keyId"]

field.pop("keyId", None)

Copilot · 2025-08-14T00:24:29Z

django_mongodb_backend/schema.py

+        else:
+            db.create_collection(db_table)
+
+    def _get_encrypted_fields_map(self, model, client, create_data_keys=False):


The method signature is inconsistent with its usage. At line 448, it's called with auto_encryption_opts as the third parameter, but the method signature expects create_data_keys as the third parameter. This will cause incorrect behavior.

Copilot · 2025-08-14T00:24:29Z

docs/source/topics/queryable-encryption.rst

+
+    class Patient(models.Model):
+        name = models.CharField(max_length=255)
+        ssn = models.EncryptedCharField(max_length=11, queries={"queryType": "equality"})


Incorrect import reference. Should be EncryptedCharField from django_mongodb_backend.fields, not models.EncryptedCharField. The import at line 71 shows the correct import path.

Suggested change

ssn = models.EncryptedCharField(max_length=11, queries={"queryType": "equality"})

ssn = EncryptedCharField(max_length=11, queries={"queryType": "equality"})

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Aug 12, 2025

View reviewed changes

docs/source/ref/django-admin.rst Outdated Show resolved Hide resolved

timgraham reviewed Aug 12, 2025

View reviewed changes

docs/source/ref/settings.rst Outdated Show resolved Hide resolved

timgraham reviewed Aug 12, 2025

View reviewed changes

docs/source/howto/queryable-encryption.rst Outdated Show resolved Hide resolved

timgraham reviewed Aug 12, 2025

View reviewed changes

docs/source/howto/queryable-encryption.rst Outdated Show resolved Hide resolved

aclark4life force-pushed the INTPYTHON-527 branch from 25b7e8b to 2469f84 Compare August 12, 2025 18:47

aclark4life commented Aug 13, 2025

View reviewed changes

.evergreen/run-tests.sh Show resolved Hide resolved

aclark4life commented Aug 13, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

aclark4life force-pushed the INTPYTHON-527 branch from cd8aa5c to 62db35a Compare August 13, 2025 14:31

aclark4life and others added 9 commits August 13, 2025 12:08

INTPYTHON-527 Add Queryable Encryption support

529db96

Code review fixes (1/x)

54f8447

QE Doc flow - Three top level (or close to it) docs - How to guide on QE configuration - QE field guide (moved out of model field ref to its own section) - Topic guide on QE usage

Code review fixes (2/2)

1294250

few edits

4c0be1b

fix code highlighting in docs

cd7c52d

Code review fixes (1/x)

08eb6c1

- Planning to finish QE setup in the how-to doc then switch back to the topic doc to explain the workflow.

Code review fixes (2/x)

7db2c94

Code review fixes (4/x)

cd035eb

- Configuring the encrypted_fields_map section

aclark4life force-pushed the INTPYTHON-527 branch from 21c237e to cd035eb Compare August 13, 2025 16:09

aclark4life added 2 commits August 13, 2025 12:47

Code review fixes (5/x)

621c44e

- Configuring the Automatic Encryption Shared Library

Code review fixes (6/6)

4fe6a35

aclark4life requested review from timgraham and Copilot August 14, 2025 00:14

This comment was marked as resolved.

Sign in to view

aclark4life and others added 4 commits August 13, 2025 20:16

Update django_mongodb_backend/management/commands/showencryptedfields…

1c57034

…map.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/source/topics/queryable-encryption.rst

25c5217

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot review fixes

d3bb833

Update docs/source/topics/queryable-encryption.rst

94a3b48

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

aclark4life requested a review from Copilot August 14, 2025 00:23

Copilot AI reviewed Aug 14, 2025

View reviewed changes

	master_key = self.connection.settings_dict.get("KMS_CREDENTIALS").get(kms_provider)
	master_key = self.connection.settings_dict.get("KMS_CREDENTIALS", {}).get(kms_provider)



	# This function is intended to be monkey-patched as a method of ConnectionRouter.

	ssn = models.EncryptedCharField(max_length=11, queries={"queryType": "equality"})
	ssn = EncryptedCharField(max_length=11, queries={"queryType": "equality"})

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Uh oh!

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

timgraham Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

aclark4life Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

aclark4life commented Jun 27, 2025 •

edited

Loading