forked from postgres/postgres
-
Notifications
You must be signed in to change notification settings - Fork 0
Dev/pytest #26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jchampio
wants to merge
6
commits into
master
Choose a base branch
from
dev/pytest
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Dev/pytest #26
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5ad83cb
to
16da32f
Compare
db7f82a
to
bc51b26
Compare
2d420c0
to
b9ca449
Compare
...to make it obvious when they've been enabled. prove is added to the executables list for good measure. TODO: does Autoconf need something similar? Per complaint by Peter Eisentraut.
86f7f9e
to
39f94a0
Compare
Specify --enable-pytest/-Dpytest=enabled at configure time. This contains no Postgres test logic -- it is just a "vanilla" pytest skeleton. I've written a custom pgtap output plugin, used by the Meson mtest runner, to fully control what we see during CI test failures. The pytest-tap plugin would have been preferable, but it's now in maintenance mode, and it has problems with accidentally suppressing important collection failures. test_something.py is intended to show a sample failure in the CI. TODOs: - check_pytest.py has a known issue: it checks the Python interpreter linked to PL/Python rather than the Python interpreter in use by pytest. Perhaps the check script should just go away? - OpenBSD has an ANSI-related terminal bug, but I'm not sure if the bug is in Cirrus, the image, pytest, Python, or readline. The TERM envvar is unset to work around it. If this workaround is removed, a bad ANSI escape is inserted into the pgtap output and mtest is unable to parse it. - The Chocolatey CI setup is subpar. Need to find a way to bless the dependencies in use rather than pulling from pip... or maybe that will be done by the image baker.
This is a sample client-only test suite. It tests some handshake failures against a mock server, as well as a full SSL handshake + empty query + response. pyca/cryptography is added as a new package dependency. Certificates for testing are generated on the fly. The `pg` test package contains some helpers and fixtures (as well as some self-tests for more complicated behavior). Of note: - pg.require_test_extra() lets you mark a test/class/module as skippable if PG_TEST_EXTRA does not contain the necessary strings. - pg.remaining_timeout() is a function which can be repeatedly called to determine how much of the PG_TEST_TIMEOUT_DEFAULT remains for the current test item. - pg.libpq is a fixture that wraps libpq.so in a more friendly, but still low-level, ctypes FFI. Allocated resources are unwound and released during test teardown. The mock design is threaded: the server socket is listening on a background thread, and the test provides the server logic via a callback. There is some additional work still needed to make this production-ready; see the notes for _TCPServer.background(). (Currently, an exception in the wrong place could result in a hang-until-timeout rather than an immediate failure.) TODOs: - local_server and tcp_server_class are nearly identical and should share code. - fix exception-related timeouts for .background() - figure out the proper use of "session" vs "module" scope - ensure that pq.libpq unwinds (to close connections) before tcp_server; see comment in test_server_with_ssl_disabled()
In the same vein as the previous commit, this is a server-only test suite operating against a mock client. The test itself is a heavily parameterized check for direct-SSL handshake behavior, using a combination of "standard" and "custom" certificates via the certs fixture. installcheck is currently unsupported, but the architecture has some extension points that should make it possible later. For now, a new server is always started for the test session. New session-level fixtures have been added which probably need to migrate to the `pg` package. Of note: - datadir points to the server's data directory - sockdir points to the server's UNIX socket/lock directory - server_instance actually inits and starts a server via the pg_ctl on PATH (and could eventually point at an installcheck target) Wrapping these session-level fixtures is pg_server[_session], which provides APIs for configuration changes that unwind themselves at the end of fixture scopes. There's also an example of nested scopes, via pg_server_session.subcontext(). Many TODOs remain before we're on par with Test::Cluster, but this should illustrate my desired architecture pretty well. Windows currently uses SCRAM-over-UNIX for the admin account rather than SSPI-over-TCP. There's some dead Win32 code in pg.current_windows_user, but I've kept it as an illustration of how a developer might write such code for SSPI. I'll probably remove it in a future patch version. TODOs: - port more server configuration behavior from PostgreSQL::Test::Cluster - decide again on "session" vs. "module" scope for server fixtures - improve remaining_timeout() integration with socket operations; at the moment, the timeout resets on every call rather than decrementing
Should make it easier to control the test cycle time for Cirrus. Add the desired suites (remembering `--suite setup`!) to the top-level envvar.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.