Document design decisions for new test framework #3214
Conversation
There was a problem hiding this comment.
We should also include somewhere a section on running tests manually/locally. That should be as straight forward as possible.
Ideally both to run the tests in a local chroot/VM and running tests on a cloud VM should be straight forward as long as one brings their own credentials. So the openTofu logic etc must be easily usable outside of the github automations without complex local setups. (both on Linux and macOS dev workstations)
|
|
||
| These approaches were not adopted due to the following limitations: | ||
| - **Software Availability:** Container runtimes and systemd are not present in all target environments. | ||
| - **Permission Requirements:** Both methods require elevated privileges, which may not be feasible or desirable in production or restricted systems. |
There was a problem hiding this comment.
The permissions/privileges concern is not just, even not primarily, about production systems. One of the main reasons to aim for privilege less test setup and execution is to allow running tests as a developer (against a locally build artifact) without requiring any privileges beyond what's needed to build (so unprivileged user namespaces only).
|
|
||
| ## Decision | ||
|
|
||
| The redesigned test framework will treat the system under test as strictly read-only. Tests must not modify system state, install packages, enable services, or change configuration. The framework itself will not require SSH setup or any other mutation of the target system. All test logic must operate without side effects, ensuring that the system remains unchanged before, during, and after test execution. |
There was a problem hiding this comment.
While in principle, yes, we do want to avoid having tests modify the system state; this cannot always be avoided for all tests (e.g. testing of loading kernel modules works). So instead we should build the tests such that tests that modify global system state MUST be clearly marked via a pytest marker and skipped unless the test framework is run with an explicit arg to allow modifications.
These tests would then only ever be run on ephemeral targets, such as local test only VMs or during the platform tests.
|
|
||
| - **Distribution Mechanisms:** The test suite may be delivered via scp, cloud-init/user_data, OCI registry/artifact, image attach, or other platform-specific methods. The suite will be packaged as a relocatable tarball or directory, and may be built on demand or pulled as a build artifact. | ||
| - **Cloud Provider Support:** Image formats and deployment workflows will be adapted for each provider (e.g., raw, vhd, qcow2), with research into automation and API integration for disk/image attachment. | ||
| - **Reporting:** Test results will be exported in a [diki](https://github.com/gardener/diki)-compatible format as part of the MVP, enabling integration with external systems and dashboards. Additional formats (e.g., JUnit XML) may be supported as needed. |
There was a problem hiding this comment.
Did we discuss/agree on the test result output already? My last state was that we would primarily aim for human readable test run logs with other formats as a side channel output, but I guess these details are still open for discussion.
There was a problem hiding this comment.
Diki requirement is mentioned here #3156 (comment)
I mean, it could be designed in a way that new outputs formats can be extended without rewriting the tests/frameworks, so maybe this should really not be specified here
I tried to address this here https://github.com/gardenlinux/gardenlinux/pull/3214/files#diff-a22547a436b6499cc9954c40fe0b182df42a88cef14f1e1a89fab59a84105d69R28 |
|
@nkraetzschmar thanks for your review, I've addressed the comments |
|
@nkraetzschmar should be reviewable now where the initial framework is merged |
| - **Distribution Mechanisms:** The test suite may be delivered via scp, cloud-init/user_data, OCI registry/artifact, image attach, or other platform-specific methods. The suite will be packaged as a relocatable tarball or directory, and may be built on demand or pulled as a build artifact. | ||
| - **Cloud Provider Support:** Image formats and deployment workflows will be adapted for each provider (e.g., raw, vhd, qcow2), with research into automation and API integration for disk/image attachment. | ||
| - **Reporting:** Test output will be flexible and allow custom formats in a plugin-based system, so that new formats are easy to add. The default output will be a human-readable text format, machine readable outputs such as a [diki](https://github.com/gardener/diki)-compatible format or JUnit xml output may be added later. | ||
| - **Backchannel for Logs:** Mechanisms such as scp, custom APIs, or direct S3 uploads will be explored for retrieving logs and results from the system under test. |
There was a problem hiding this comment.
I think this part should be formulated more general. Such as: As for reporting itself the delivery of the logs and results will be implemented in a plugin-based approach to suite execution scenarios best.
| - **Portability:** The test suite can run in containers, chroots, VMs, bare metal, and production systems. | ||
| - **Flexibility:** Multiple deployment mechanisms are supported; the framework is not tied to a specific transport or runtime. | ||
| - **Maintainability:** The framework is easier to reason about and maintain, as tests run in a predictable, local context. | ||
| - **Reporting:** Output can be collected via stdout/stderr, persisted as JUnit XML, or exported in other formats. |
There was a problem hiding this comment.
It is not clear to me how the output is collected via stdout/stderr. Sure, there must be a command dispatch to run the tests, but to me it seems like the new framework minimizes shell interaction. I'm probably missing something here, but maybe you could add some details to make things clear
Based on #3156 and #3159
This PR documents the design decisions for the new testing framework.