-
-
Notifications
You must be signed in to change notification settings - Fork 32.6k
Refs #34986 -- Confirmed support for PyPy 3.10 with PostgreSQL. #17500
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
f521026
to
a598986
Compare
Thanks a lot for doing this! Could you also include something like: diff --git a/tox.ini b/tox.ini
index 2e0a3b421a..c635a129b2 100644
--- a/tox.ini
+++ b/tox.ini
@@ -26,7 +26,7 @@ setenv =
PYTHONDONTWRITEBYTECODE=1
deps =
-e .
- py{3,310,311,312}: -rtests/requirements/py3.txt
+ py{3,310,311,312,py3}: -rtests/requirements/py3.txt
postgres: -rtests/requirements/postgres.txt
mysql: -rtests/requirements/mysql.txt
oracle: -rtests/requirements/oracle.txt |
@mgorny Sure. I'll see if there is anything else that makes sense while I'm at it. |
Currently down to 2 failures and 1 error running on SQLite 😎
The model_forms.tests.OtherModelFormTests.test_prefetch_related_queryset one is weird...On CPython we get:
On PyPy we get:
Update: Aha! It turns out that PyPy isn't calling On CPython: >>> class CustomIterator:
... def __iter__(self):
... yield from range(3)
... def __len__(self):
... print("CustomIterator.__len__()")
... return 3
...
>>> tuple(CustomIterator())
CustomIterator.__len__()
(0, 1, 2) On PyPy: >>>> class CustomIterator:
.... def __iter__(self):
.... yield from range(3)
.... def __len__(self):
.... print("CustomIterator.__len__()")
.... return 3
....
>>>> tuple(CustomIterator())
(0, 1, 2) |
For I get the following traceback for CPython: Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 277, in __call__
return call_result.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 353, in main_wrap
result = await self.awaitable(*args, **kwargs)
File "/tests/django/django/test/utils.py", line 434, in inner
return await func(*args, **kwargs)
File "/tests/django/tests/auth_tests/test_auth_backends.py", line 773, in test_aauthenticate_sensitive_variables
await aauthenticate(
File "/tests/django/django/contrib/auth/__init__.py", line 99, in aauthenticate
return await sync_to_async(authenticate)(request, **credentials)
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 479, in __call__
ret: _R = await loop.run_in_executor(
File "/usr/local/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
return func(*args, **kwargs)
File "/tests/django/django/views/decorators/debug.py", line 78, in sensitive_variables_wrapper
return func(*func_args, **func_kwargs)
File "/tests/django/django/contrib/auth/__init__.py", line 79, in authenticate
user = backend.authenticate(request, **credentials)
File "/tests/django/django/views/decorators/debug.py", line 78, in sensitive_variables_wrapper
return func(*func_args, **func_kwargs)
File "/tests/django/tests/auth_tests/test_auth_backends.py", line 717, in authenticate
raise TypeError
TypeError And the following for PyPy: Traceback (most recent call last):
File "/opt/pypy/lib/pypy3.10/site-packages/asgiref/sync.py", line 277, in __call__
return call_result.result()
File "/opt/pypy/lib/pypy3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/opt/pypy/lib/pypy3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/opt/pypy/lib/pypy3.10/asyncio/tasks.py", line 304, in __wakeup
future.result()
File "/opt/pypy/lib/pypy3.10/asyncio/futures.py", line 201, in result
raise self._exception.with_traceback(self._exception_tb)
File "/opt/pypy/lib/pypy3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/pypy/lib/pypy3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
return func(*args, **kwargs)
File "/opt/pypy/lib/pypy3.10/_contextvars.py", line 44, in run
return callable(*args, **kwargs)
File "/tests/django/django/views/decorators/debug.py", line 78, in sensitive_variables_wrapper
return func(*func_args, **func_kwargs)
File "/tests/django/django/contrib/auth/__init__.py", line 79, in authenticate
user = backend.authenticate(request, **credentials)
File "/tests/django/django/views/decorators/debug.py", line 78, in sensitive_variables_wrapper
return func(*func_args, **func_kwargs)
File "/tests/django/tests/auth_tests/test_auth_backends.py", line 717, in authenticate
raise TypeError
TypeError I suspect we get away with all the
We could extend Blergh. Not sure where to go from here. |
cf28a73
to
aa41210
Compare
CC @mattip, @cfbolz; could you particularly take a look at #17500 (comment) and advise? |
I'm currently fixing the pattern matching bug in pypy. @ngnpope missing context to help here a little bit. is the problem that CPython's |
Thanks for the pattern matching fix, @cfbolz. I'll give that a whirl when the next nightly is out 🎉 I'll try and work out a little more detail of what is going on and get back to you regarding this last test failure. |
aa41210
to
fbc0dda
Compare
fbc0dda
to
a6bbbd0
Compare
@cfbolz Sorry for the delay getting back to you. This is the last failure, so we're making progress! I've looked at the tracebacks and commented out the common bits between CPython and PyPy. Here is the one for CPython: # Traceback (most recent call last):
# File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 277, in __call__
# return call_result.result()
# File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
# return self.__get_result()
# File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
# raise self._exception
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 353, in main_wrap
result = await self.awaitable(*args, **kwargs)
File "/django/source/django/test/utils.py", line 434, in inner
return await func(*args, **kwargs)
File "/django/source/tests/auth_tests/test_auth_backends.py", line 773, in test_aauthenticate_sensitive_variables
await aauthenticate(
File "/django/source/django/contrib/auth/__init__.py", line 99, in aauthenticate
return await sync_to_async(authenticate)(request, **credentials)
File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 479, in __call__
ret: _R = await loop.run_in_executor(
# File "/usr/local/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
# result = self.fn(*self.args, **self.kwargs)
# File "/usr/local/lib/python3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
# return func(*args, **kwargs)
# File "/django/source/django/views/decorators/debug.py", line 73, in sensitive_variables_wrapper
# return func(*func_args, **func_kwargs)
# File "/django/source/django/contrib/auth/__init__.py", line 79, in authenticate
# user = backend.authenticate(request, **credentials)
# File "/django/source/django/views/decorators/debug.py", line 73, in sensitive_variables_wrapper
# return func(*func_args, **func_kwargs)
# File "/django/source/tests/auth_tests/test_auth_backends.py", line 717, in authenticate
# raise TypeError
# TypeError And the one for PyPy: # Traceback (most recent call last):
# File "/opt/pypy/lib/pypy3.10/site-packages/asgiref/sync.py", line 277, in __call__
# return call_result.result()
# File "/opt/pypy/lib/pypy3.10/concurrent/futures/_base.py", line 451, in result
# return self.__get_result()
# File "/opt/pypy/lib/pypy3.10/concurrent/futures/_base.py", line 403, in __get_result
# raise self._exception
File "/opt/pypy/lib/pypy3.10/asyncio/tasks.py", line 304, in __wakeup
future.result()
File "/opt/pypy/lib/pypy3.10/asyncio/futures.py", line 201, in result
raise self._exception.with_traceback(self._exception_tb)
# File "/opt/pypy/lib/pypy3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
# result = self.fn(*self.args, **self.kwargs)
# File "/opt/pypy/lib/pypy3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
# return func(*args, **kwargs)
File "/opt/pypy/lib/pypy3.10/_contextvars.py", line 44, in run
return callable(*args, **kwargs)
# File "/django/source/django/views/decorators/debug.py", line 73, in sensitive_variables_wrapper
# return func(*func_args, **func_kwargs)
# File "/django/source/django/contrib/auth/__init__.py", line 79, in authenticate
# user = backend.authenticate(request, **credentials)
# File "/django/source/django/views/decorators/debug.py", line 73, in sensitive_variables_wrapper
# return func(*func_args, **func_kwargs)
# File "/django/source/tests/auth_tests/test_auth_backends.py", line 717, in authenticate
# raise TypeError
# TypeError I'm not sure why there is a difference with the first block - perhaps something different happens in PyPy when stitching frames together in relation to async code? I seem to be able to get some of that output if I fiddle with the way that exceptions are caught, etc. But I think we don't need to worry about this bit too much. The issue is, as you thought, that functools.partial(<function authenticate at 0x00007fec19875b00>, None, username='testusername', password='mypassword'``` So we could add An alternative is to add something to always mask diff --git a/django/views/debug.py b/django/views/debug.py
index c1265bfe6b..b280f96c66 100644
--- a/django/views/debug.py
+++ b/django/views/debug.py
@@ -195,6 +195,14 @@ class SafeExceptionReporterFilter:
multivaluedict[param] = self.cleansed_substitute
return multivaluedict
+ def get_cleansed_partial(self, request, value):
+ """Replace all arguments to partial functions."""
+ return type(value)(
+ value.func,
+ *(None if v is None else self.cleansed_substitute for v in value.args),
+ **{key: self.cleansed_substitute for key in value.keywords},
+ )
+
def get_post_parameters(self, request):
"""
Replace the values of POST parameters marked as sensitive with
@@ -234,6 +242,10 @@ class SafeExceptionReporterFilter:
if is_multivalue_dict:
# Cleanse MultiValueDicts (request.POST is the one we usually care about)
value = self.get_cleansed_multivaluedict(request, value)
+
+ if isinstance(value, (functools.partial, functools.partialmethod)):
+ value = self.get_cleansed_partial(request, value)
+
return value
def get_traceback_frame_variables(self, request, tb_frame):
This fixed the test for me. What are your thoughts on that @felixxm? |
a6bbbd0
to
d3281cf
Compare
I'm skeptical about fixing something in Django that is not Django's fault. |
PyPy has a |
@mattip That sounds perfect! If you could add that it'd be great. (I've also provided the minimal reproducers you requested above.) |
I hid
Thanks, I opened python/cpython#112451 |
Thanks @mattip. Could you also merge that into the pypy3.10 branch please? |
Pushed to py3.10 too |
e469a35
to
c822214
Compare
Regarding using PostgreSQL, with the fix in psycopg/psycopg#686, and the non-binary version I only get the following errors:
For which we could do the following 😖 diff --git a/tests/backends/postgresql/test_server_side_cursors.py b/tests/backends/postgresql/test_server_side_cursors.py
index 705e798c23..712fff8371 100644
--- a/tests/backends/postgresql/test_server_side_cursors.py
+++ b/tests/backends/postgresql/test_server_side_cursors.py
@@ -1,3 +1,4 @@
+import gc
import operator
import unittest
from collections import namedtuple
@@ -5,6 +6,7 @@ from contextlib import contextmanager
from django.db import connection, models
from django.test import TestCase
+from django.utils.version import PYPY
from ..models import Person
@@ -88,9 +90,12 @@ class ServerSideCursorsPostgres(TestCase):
persons = Person.objects.iterator()
next(persons) # Open a server-side cursor
del persons
+ if PYPY:
+ gc.collect()
cursors = self.inspect_cursors()
self.assertEqual(len(cursors), 0)
+ @unittest.skipIf(PYPY, reason="Forced garbage collection breaks test.")
def test_server_side_cursors_setting(self):
with self.override_db_setting(DISABLE_SERVER_SIDE_CURSORS=False):
persons = Person.objects.iterator()
|
56cf1ee
to
3a7edd1
Compare
Updated using an alternative unpickleable object as mentioned in #17500 (comment) and added some fixes for PostgreSQL. I've also added a commit to extend the scheduled testing to include PostgreSQL and add the ability to execute tests for PyPy using a label on a PR. |
I don't think we want more CI tests for PyPy. I don't see much value in labeled CI tests for PyPy, because how could we decide if something might not be compatible with PyPy? Daily builds seems to be sufficient in such rare cases. |
Ok. I'll revert that one to only add a daily run with PostgreSQL. |
I merged 4 commits from this PR, please rebase. |
Thanks. Rebased. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ngnpope Thanks 👍
Thanks a lot for all your hard work! Will these changes make it to 5.0.x? |
Unfortunately not. |
@mgorny Most of the changes were to Django's tests to handle minor implementation differences. If you're not running those, then there are very few changes. Note that you'll need to run using a nightly build of PyPy right now as there were a few tweaks there over PyPy v7.3.13. And we've only tested for SQLite and PostgreSQL (which requires I think the only one that'll stop PyPy working for 5.0 is 5f9e5c1 to fix a crash in serializing for migrations. (This was related to something newly added in 5.0, but arguably wasn't a release blocker because PyPy support had stagnated anyway.) And, if you want to use Django's autoreloader while using Python's |
No problem. I've been able to backport them easily to 5.0, 4.2 and 4.1 branches in Gentoo. Thanks a lot, again! |
ticket-34986