Skip to content

Conversation

jchampio
Copy link
Owner

No description provided.

@jchampio jchampio force-pushed the dev/pytest branch 14 times, most recently from 5ad83cb to 16da32f Compare August 25, 2025 21:34
@jchampio jchampio force-pushed the dev/pytest branch 16 times, most recently from db7f82a to bc51b26 Compare September 3, 2025 21:56
@jchampio jchampio force-pushed the dev/pytest branch 21 times, most recently from 2d420c0 to b9ca449 Compare September 5, 2025 20:02
...to make it obvious when they've been enabled. prove is added to the
executables list for good measure.

TODO: does Autoconf need something similar?

Per complaint by Peter Eisentraut.
@jchampio jchampio force-pushed the dev/pytest branch 2 times, most recently from 86f7f9e to 39f94a0 Compare September 9, 2025 20:58
Specify --enable-pytest/-Dpytest=enabled at configure time. This
contains no Postgres test logic -- it is just a "vanilla" pytest
skeleton.

I've written a custom pgtap output plugin, used by the Meson mtest
runner, to fully control what we see during CI test failures. The
pytest-tap plugin would have been preferable, but it's now in
maintenance mode, and it has problems with accidentally suppressing
important collection failures.

test_something.py is intended to show a sample failure in the CI.

TODOs:
- check_pytest.py has a known issue: it checks the Python interpreter
  linked to PL/Python rather than the Python interpreter in use by
  pytest. Perhaps the check script should just go away?
- OpenBSD has an ANSI-related terminal bug, but I'm not sure if the bug
  is in Cirrus, the image, pytest, Python, or readline. The TERM envvar
  is unset to work around it. If this workaround is removed, a bad ANSI
  escape is inserted into the pgtap output and mtest is unable to parse
  it.
- The Chocolatey CI setup is subpar. Need to find a way to bless the
  dependencies in use rather than pulling from pip... or maybe that will
  be done by the image baker.
This is a sample client-only test suite. It tests some handshake
failures against a mock server, as well as a full SSL handshake + empty
query + response.

pyca/cryptography is added as a new package dependency. Certificates for
testing are generated on the fly.

The `pg` test package contains some helpers and fixtures (as well as
some self-tests for more complicated behavior). Of note:

- pg.require_test_extra() lets you mark a test/class/module as skippable
  if PG_TEST_EXTRA does not contain the necessary strings.

- pg.remaining_timeout() is a function which can be repeatedly called to
  determine how much of the PG_TEST_TIMEOUT_DEFAULT remains for the
  current test item.

- pg.libpq is a fixture that wraps libpq.so in a more friendly, but
  still low-level, ctypes FFI. Allocated resources are unwound and
  released during test teardown.

The mock design is threaded: the server socket is listening on a
background thread, and the test provides the server logic via a
callback. There is some additional work still needed to make this
production-ready; see the notes for _TCPServer.background(). (Currently,
an exception in the wrong place could result in a hang-until-timeout
rather than an immediate failure.)

TODOs:
- local_server and tcp_server_class are nearly identical and should
  share code.
- fix exception-related timeouts for .background()
- figure out the proper use of "session" vs "module" scope
- ensure that pq.libpq unwinds (to close connections) before tcp_server;
  see comment in test_server_with_ssl_disabled()
In the same vein as the previous commit, this is a server-only test
suite operating against a mock client. The test itself is a heavily
parameterized check for direct-SSL handshake behavior, using a
combination of "standard" and "custom" certificates via the certs
fixture.

installcheck is currently unsupported, but the architecture has some
extension points that should make it possible later. For now, a new
server is always started for the test session.

New session-level fixtures have been added which probably need to
migrate to the `pg` package. Of note:

- datadir points to the server's data directory
- sockdir points to the server's UNIX socket/lock directory
- server_instance actually inits and starts a server via the pg_ctl on
  PATH (and could eventually point at an installcheck target)

Wrapping these session-level fixtures is pg_server[_session], which
provides APIs for configuration changes that unwind themselves at the
end of fixture scopes. There's also an example of nested scopes, via
pg_server_session.subcontext(). Many TODOs remain before we're on par
with Test::Cluster, but this should illustrate my desired architecture
pretty well.

Windows currently uses SCRAM-over-UNIX for the admin account rather than
SSPI-over-TCP. There's some dead Win32 code in pg.current_windows_user,
but I've kept it as an illustration of how a developer might write such
code for SSPI. I'll probably remove it in a future patch version.

TODOs:
- port more server configuration behavior from PostgreSQL::Test::Cluster
- decide again on "session" vs. "module" scope for server fixtures
- improve remaining_timeout() integration with socket operations; at the
  moment, the timeout resets on every call rather than decrementing
Should make it easier to control the test cycle time for Cirrus. Add the
desired suites (remembering `--suite setup`!) to the top-level envvar.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant