Update test suite #2181

ntoll · 2024-09-23T10:31:23Z

Description

This is a major refactor of the Python related testing of PyScript.

Detail - PyScript lives in two worlds: Python and JavaScript.

We have lots of JS tests and some clunky Python tests (originally run by playwright with the Python tests expressed as strings!). This refactor makes use of uPyTest (a PyTest inspired test framework that runs in PyScript in the browser) to run tests expressed as actual Python code in PyScript.

Changes

All the old Python integration tests have been deleted, with the following caveats:

Tests that used to exercise actual Python code inside PyScript have been copied over / refactored into the new Python integration tests.
Tests that check Pyodide or MicroPython have been removed (we should trust our upstream).
Tests that check PyScript is correctly running in the browser duplicate many of our existing JS tests (with JS tests being the more appropriate place to run these), have also been deleted.

I have also added tests for all the parts of the current Pythonic pyscript namespace available within PyScript. In addition to checking functionality, some of these tests are simple "sanity checks" to ensure features (such as pyscript.config or pyscript.RUNNING_IN_WORKER) are working.

Checklist

All tests pass locally
I have updated CHANGELOG.md
I have created documentation for this(if applicable)

…butes.

for more information, see https://pre-commit.ci

…butes.

for more information, see https://pre-commit.ci

… update-test-suite

for more information, see https://pre-commit.ci

…t step, since this is already built into the build step.

for more information, see https://pre-commit.ci

WebReflection · 2024-09-24T16:47:48Z

pyscript.core/package.json

@@ -58,7 +58,7 @@
        "@ungap/with-resolvers": "^0.1.0",
        "@webreflection/idb-map": "^0.3.1",
        "basic-devtools": "^0.1.6",
-        "polyscript": "^0.15.6",
+        "polyscript": "^0.15.7",


Something went wrong with the previous MR ... this should be 0.15.8 instead! That being said, it's not clear to me why this file shows up with changes already merged in main ... I think you should rebase properly then change only this version of polyscript (or I can file a new MR).

Hmm... this update clearly hasn't landed in main: https://github.com/pyscript/pyscript/blob/main/pyscript.core/package.json#L61 (I still see 0.15.7)

yes, it doesn't mean it cannot land in here, it's the same code more or less, just one less PR to approve ;-)

this is still outdated ... can you please make it 0.15.8 ?

Done. Dammit - I knew I'd missed something,

WebReflection · 2024-09-24T16:50:42Z

pyscript.core/tests/integration/python/helper.js

+// terminal=0 to NOT have a terminal
+const terminal = qs.get('terminal') !== '0';
+
+// worker=1 to have a worker


I find these two a bit confusing ... we could as well just use has('terminal') or has('worker') ... not sure why we need =0 on terminal and =1 on worker ... this is a nit though, still maybe you agree.

This has more to do with my shonky JS "skillz". 😉 I'll fix this up... I agree with your proposed change. It's much more readable. 👍

it was probably me suggesting these two but that's because I forgot that ?terminal&worker are perfectly valid too as query strings parameters

WebReflection · 2024-09-24T16:53:48Z

pyscript.core/tests/integration/python/index.html

+<!DOCTYPE html>
+<html lang="en">
+    <head>
+        <script src="mini-coi.js" scope="./"></script>


as much as I think mini-coi validated in here could make some sense, I don't think we should test our code with a service worker ... our PyScript .com doesn't need or use one, so that test should likely be a part and very simple (if you can't change window.document.body.textContent from a worker, that's a failing Service Worker, nothing else literally changes).

I am saying this because behind the scene there is also sabayon SW approach but that's tested and code covered in sabayon already so we should provide "production code" (imho) ... your call (and mini-coi wasn't needed before)

WebReflection · 2024-09-24T16:55:55Z

pyscript.core/tests/ws/README.md

@@ -0,0 +1,34 @@
+# PyScript Test Suite
+
+The test suite is a bit of a mess. We need to fix it.


Isn't the purpose of this MR to fix this status? Just wondering if this file needs any update or if we should just remove (and I'll follow up with more cleanup around JS, if needed).

ntoll · 2024-09-25T08:55:29Z

@WebReflection thanks for the feedback. Yes - this is very much still a WiP and I'll fix up things later today (currently about to step into a meeting).

I'll also move the javascript-integration to the right (new) place. We need to discuss where the other tests live. I assume some will go in "tests/manual" and others could be converted to JS integration tests.

WebReflection · 2024-09-25T08:58:25Z

@ntoll

I assume some will go in "tests/manual" and others could be converted to JS integration tests.

integration tests that are in integration should remain integration tests ... manual tests should remain manual. Where we move these doesn't change their nature but also I don't think we need to do everything in this MR or it will delay for very little gain in terms of changes (because the old slow tests suit still runs on each commit).

for more information, see https://pre-commit.ci

ntoll · 2024-09-25T13:06:31Z

@WebReflection OK. I've moved things around as we discussed so that:

tests/python - Python test suite.
tests/javascript - Formerly the js-integration tests.
tests/manual - remain the same, in that they contain "manual" tests.
TODO: All the other directories in tests/ should be moved to tests/manual or tests/javascript or deleted, depending on what they do.
I've also updated the READMEs, as needed.

HEADS UP: I've noticed that the MicroPython + workers JavaScript test sometimes times out. Somehow it appears to get stuck in some way.

WebReflection · 2024-09-25T15:14:52Z

@ntoll

HEADS UP: I've noticed that the MicroPython + workers JavaScript test sometimes times out. Somehow it appears to get stuck in some way.

sometimes it's a CDN issue so it takes long time to even bootstrap the interpreter ... bear in mind that's not our own CDN where we serve PyScript, I am not sure how to tackle that but yes, it happened to me too and maybe we should investigate more if it's instead a matter of async checks thta are badly organized. If it keeps happening (very sporadic in my local env) it should get high priority.

WebReflection · 2024-09-25T15:21:40Z

@ntoll actually ... I see you just increased the timeout ... I am not sure that's the right fix but if it makes tests less flaky I am OK with it, hence my approval.

ntoll added 8 commits September 23, 2024 17:37

WiP pyscript.web tests pass with upytest.

6650809

Test refactoring. WiP

ec20298

Completed refactor of old integration tests to new Python tests.

762ae19

Added comprehensive test suite for Python based pyodide module.

b35fb8f

Black.

806e574

Rebuld and check websocket attribute assignment via init and as attri…

177cbb9

…butes.

Update when tests to allow for worker round trips.

091da6e

Post build

4a6c5fa

ntoll force-pushed the update-test-suite branch from 4a39ff6 to 4a6c5fa Compare September 23, 2024 16:53

pre-commit-ci bot and others added 20 commits September 23, 2024 16:54

[pre-commit.ci] auto fixes from pre-commit.com hooks

790fd26

for more information, see https://pre-commit.ci

Fixed config issue via polyscript update (#2182)

ae66d13

WiP pyscript.web tests pass with upytest.

e3c572d

Test refactoring. WiP

aa83e6c

Completed refactor of old integration tests to new Python tests.

263f1fc

Added comprehensive test suite for Python based pyodide module.

0eb8c43

Black.

6c81b5f

Rebuld and check websocket attribute assignment via init and as attri…

43f25fd

…butes.

Update when tests to allow for worker round trips.

344e54a

Post build

f24a2cf

[pre-commit.ci] auto fixes from pre-commit.com hooks

6e52918

for more information, see https://pre-commit.ci

Merge branch 'update-test-suite' of github.com:pyscript/pyscript into…

06216ac

… update-test-suite

Fix a couple of timing issues in display and web tests.

abc3a3b

[pre-commit.ci] auto fixes from pre-commit.com hooks

0d7ef9d

for more information, see https://pre-commit.ci

Black

57e023e

[pre-commit.ci] auto fixes from pre-commit.com hooks

58d905d

for more information, see https://pre-commit.ci

Add integration tests to Makefile (and CI)

7da2ba6

Remove un-needed upload action.

f826d0b

Ensure fails are properly logged as an array. Remove the explicit tes…

af6cc2b

…t step, since this is already built into the build step.

[pre-commit.ci] auto fixes from pre-commit.com hooks

aa70295

for more information, see https://pre-commit.ci

WebReflection reviewed Sep 24, 2024

View reviewed changes

ntoll and others added 2 commits September 25, 2024 13:52

Directory reorg/refactor. Updated docs.

19bc612

[pre-commit.ci] auto fixes from pre-commit.com hooks

60622c9

for more information, see https://pre-commit.ci

ntoll marked this pull request as ready for review September 25, 2024 13:02

Bump polyscript.

c52a695

WebReflection approved these changes Sep 25, 2024

View reviewed changes

ntoll merged commit 06138bb into main Sep 25, 2024
2 checks passed

ntoll deleted the update-test-suite branch September 25, 2024 15:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update test suite #2181

Update test suite #2181

ntoll commented Sep 23, 2024 •

edited

Loading

WebReflection Sep 24, 2024

ntoll Sep 25, 2024

ntoll Sep 25, 2024

WebReflection Sep 25, 2024

WebReflection Sep 25, 2024

ntoll Sep 25, 2024

WebReflection Sep 24, 2024

ntoll Sep 25, 2024

WebReflection Sep 25, 2024

WebReflection Sep 24, 2024 •

edited

Loading

WebReflection Sep 24, 2024

ntoll commented Sep 25, 2024

WebReflection commented Sep 25, 2024

ntoll commented Sep 25, 2024

WebReflection commented Sep 25, 2024

WebReflection commented Sep 25, 2024

		@@ -0,0 +1,34 @@
		# PyScript Test Suite

		The test suite is a bit of a mess. We need to fix it.

Update test suite #2181

Update test suite #2181

Conversation

ntoll commented Sep 23, 2024 • edited Loading

Description

Changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WebReflection Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ntoll commented Sep 25, 2024

WebReflection commented Sep 25, 2024

ntoll commented Sep 25, 2024

WebReflection commented Sep 25, 2024

WebReflection commented Sep 25, 2024

ntoll commented Sep 23, 2024 •

edited

Loading

WebReflection Sep 24, 2024 •

edited

Loading