Description
Checklist
- I have verified that that issue exists against the
masterbranch of Django REST framework. - I have searched for similar issues in both open and closed tickets and cannot find a duplicate. but I think it might be related to the work here: Robust uniqueness checks. #4217
- This is not a usage question. (Those should be directed to the discussion group instead.)
- This cannot be dealt with as a third party library. (We prefer new functionality to be in the form of third party libraries where possible.)
- I have reduced the issue to the simplest possible case.
- I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)
Steps to reproduce
First: check Django behaviour
Given a simple Model with a single IntegerField, when working with SQLite (I'm running version 3.27 which is officially supported by Django) it is possible to generate an overflow error.
Model:
class OverflowModel(models.Model):
value = models.IntegerField(unique=True)Test:
def test_model(self):
OverflowModel(value=9223372036854775808).full_clean()Gives an OverflowException:
tests/test_model_serializer.py:1320: in test_model
OverflowModel(value=9223372036854775808).full_clean()
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/base.py:1217: in full_clean
self.validate_unique(exclude=exclude)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/base.py:999: in validate_unique
errors = self._perform_unique_checks(unique_checks)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/base.py:1103: in _perform_unique_checks
if qs.exists():
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/query.py:777: in exists
return self.query.has_results(using=self.db)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/sql/query.py:537: in has_results
return compiler.has_results()
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/sql/compiler.py:1114: in has_results
return bool(self.execute_sql(SINGLE))
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/sql/compiler.py:1144: in execute_sql
cursor.execute(sql, params)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/backends/utils.py:68: in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/backends/utils.py:77: in _execute_with_wrappers
return executor(sql, params, many, context)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/backends/utils.py:86: in _execute
return self.cursor.execute(sql, params)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py:396: in execute
return Database.Cursor.execute(self, query, params)
E OverflowError: Python int too large to convert to SQLite INTEGERHowever, using MaxValueValidator allows us to protect SQLite from this overflow. So if we change the model and test, then Django raises a ValidationError complaining that value is too large:
Model:
class OverflowModel(models.Model):
value = models.IntegerField(unique=True, validators=[
MaxValueValidator(9223372036854775807),
])Test:
def test_model(self):
with self.assertRaises(ValidationError):
OverflowModel(value=9223372036854775808).full_clean()In this way we can use Django to protect SQLite from the overflow - all is good at the Django level.
DRF behaviour
Now we make a ModelSerializer for this updated model with the MaxValueValidator in place and test it with the large int:
def test(self):
class TestSerializer(serializers.ModelSerializer):
class Meta:
model = OverflowModel
fields = '__all__'
serializer = TestSerializer(data={'value': 9223372036854775808})
with self.assertRaises(serializers.ValidationError):
serializer.is_valid()Expected behaviour
This test should pass. The DRF model serializer should raise a ValidationError that complains that value is too large, something like:
rest_framework.exceptions.ValidationError: {'value': [ErrorDetail(string='Ensure this value is less than or equal to 9223372036854775807.', code='max_value')]}
Actual behaviour
However the test fails with an OverflowError which arises from the uniqueness check:
tests/test_model_serializer.py:1334: in test
serializer.is_valid()
rest_framework/serializers.py:219: in is_valid
self._validated_data = self.run_validation(self.initial_data)
rest_framework/serializers.py:418: in run_validation
value = self.to_internal_value(data)
rest_framework/serializers.py:475: in to_internal_value
validated_value = field.run_validation(primitive_value)
rest_framework/fields.py:567: in run_validation
self.run_validators(value)
rest_framework/fields.py:589: in run_validators
validator(value, self)
rest_framework/validators.py:73: in __call__
if qs_exists(queryset):
rest_framework/validators.py:21: in qs_exists
return queryset.exists()
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/query.py:777: in exists
return self.query.has_results(using=self.db)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/sql/query.py:537: in has_results
return compiler.has_results()
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/sql/compiler.py:1114: in has_results
return bool(self.execute_sql(SINGLE))
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/models/sql/compiler.py:1144: in execute_sql
cursor.execute(sql, params)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/backends/utils.py:68: in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/backends/utils.py:77: in _execute_with_wrappers
return executor(sql, params, many, context)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/backends/utils.py:86: in _execute
return self.cursor.execute(sql, params)
.tox/venvs/py37-django30/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py:396: in execute
return Database.Cursor.execute(self, query, params)
E OverflowError: Python int too large to convert to SQLite INTEGERThoughts
(Ignore as appropriate)
It looks to me like the check for uniqueness happens before the field level validation that happens as a result of the MaxValueValidator because when I dump the field's validators I get:
[
<UniqueValidator(queryset=OverflowModel.objects.all())>,
<django.core.validators.MaxValueValidator object at 0x7f6743eff7f0>
]
This means that round-tripping the database happens before checking the max value. This is unexpected behaviour and I was surprised when I found it - I would expect as much validation as possible to take place before the database hit.
I also experimented with adding a field-level validate_value() function to the serializer, but it appears that run_validators() happens before that is trigged, so it's no help in preventing the overflow.