V0.6.x micropip often fails to discover package #2228

Archmonger · 2024-10-23T01:00:14Z

Checklist

I added a descriptive title
I searched for other issues and couldn't find a solution or duplication
I already searched in Google and didn't find any good information or help

What happened?

There seems to be a regression on Micropip behavior on v0.6.x where micropip will fail to install packages if the user has pyodide cached within the browser's IndexDB.

Upon clearing indexdb, the browser will then be able to install packages via micropip again.

This was tested on all Chrome, FF, and Edge on Windows 11.

This issue does not exist on 0.4.x and 0.5.x.

What browsers are you seeing the problem on? (if applicable)

Firefox, Chrome, Microsoft Edge

Console info

custom.js:182 Uncaught (in promise) Error: No known package with name 'jsonpointer==3.0.0'
    at addPackageToLoad (pyodide.asm.js:10:90432)
    at recursiveDependencies (pyodide.asm.js:10:90746)
    at loadPackage (pyodide.asm.js:10:93056)
    at initializePackageIndex (pyodide.asm.js:10:89787)

Additional Context

<py-script async config='{"packages":["reactpy==1.0.2","jsonpointer==3.0.0","ssl"],"js_modules":{"main":{"/static/reactpy_django/morphdom/morphdom-esm.js":"morphdom","/static/moment.js":"moment"}}}'></py-script>

The text was updated successfully, but these errors were encountered:

- Switch to `Bun` for building JS - Minor cleanup to pyscript renderer - Revert pyscript version to 0.5 due to [bugs](pyscript/pyscript#2228)

WebReflection · 2024-10-23T07:30:07Z

Aha, I think the cache is not smart enough to handle versioning, which is indeed a bummer.
Apologies for the regression, will try to work this out ASAP, thank you for filing this issue.

/cc @ntoll

Archmonger · 2024-10-23T07:34:47Z

Something notable is I don't believe this is related to versioning, because if I refresh the web page it proceeds to break again.

More accurately, anytime that pyodide is loaded from indexdb, then micropip does not work.

I haven't dug much deeper than that, but perhaps micropip somehow gets mangled when loading from indexdb.

WebReflection · 2024-10-24T12:35:38Z

perhaps micropip somehow gets mangled when loading from indexdb

all I hear is that this might be a Pyodide issue with the locked dependencies which would be unfortunate but right now this is just speculation as I haven't had a chance yet to look at this issue ... on my next TODO list

WebReflection · 2024-10-25T15:23:32Z

OK, I did investigate a bit around this issue and I am not sure I have much to share about it if not details:

to disable the IndexedDB related cache you can always use packages_cache = "never" in your config.toml or config.json ... looking at the presented example you should use <py-script config='{"packages_cache":"never","packages":["reactpy==1.0.2","jsonpointer==3.0.0","ssl"],"js_modules":{"main":{"/static/reactpy_django/morphdom/morphdom-esm.js":"morphdom","/static/moment.js":"moment"}}}'></py-script> ... this is not a solution to the issue, rather a "hey, until we fix it, feel free to disable this feature if it's broken for you"
there is literally nothing we do to parse versioning or stuff ... we pass the config as is and then we store it by key. This key is a sorted list of dependencies straight out of packages array without any mangling or processing attached, this is because we wanted to be sure all sort of packages combinations with or without pinned details would work
we do have integration tests around this dance and while it's true we're not trapping versions, the default seems to work as expected

Dare I summon @hoodmane around this issue because I am not really sure what is going on ... to recap:

if the packages, once sorted, produces a known key, we don't pass through micropip to install anything and we delegate pyodide to use that lockFileURL option to do the thing for us
if the key is not known, we pass through micropip manually and install all packages ... this part is covered by the fact on first run nobody has any issue whatsoever on a fresh (cleared IndexedDB) state
is it possible that pinned versions are not working as expected via micropip.freeze() ??? 🤔

WebReflection · 2024-10-25T15:39:10Z

for what is worth it, this page does the following:

<!doctype html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width,initial-scale=1.0">
        <script type="module" src="https://cdn.jsdelivr.net/npm/@pyscript/core/dist/core.js"></script>
    </head>
    <body>
      <script type="py" config='{"packages":["jsonpointer==3.0.0"]}'>
        print('Hello World')
      </script>
    </body>
</html>

if the config has versions in it, it will error on next reload/refresh: Uncaught (in promise) Error: No known package with name 'jsonpointer==3.0.0'
if the config has no version in it, it just works: <script type="py" config='{"packages":["jsonpointer"]}'> ... that would show micropip logs first time and delegate Pyodide every time after

Apologies for the second ping @hoodmane ... but wasn't the whole reason to have freeze() in micropip and pass that as lockFileURL option to Pyodide so that pinned versions would remain forever pinned in that regard?

I would be OK if once dependencies are frozen I need to remove those pinned versions out of the blue in our logic but I wonder if this is what everyone else expects too 🤔

hoodmane · 2024-10-25T16:00:44Z

Yeah we don't have any logic for parsing requirement specifiers in loadPackage. We only load locked requirements based on an exact name match. You should strip the specifier part of the requirement out yourself. I would do:

function requirementToName(requirement) {
   return requirement.split(/[^A-Za-z_-]/,1)[0];
}

Perhaps we should explain the workflow for freezing requirements somewhere in our docs...

WebReflection · 2024-10-25T16:21:15Z

@hoodmane thanks for the prompt answer but I'd raise an issue in Pyodide because to me it's not even clear where I should drop versioning .. I have the packages defined by users as unique key, and I install those packages explicitly via micropip.install(packages, { keep_going: true }); then I store micropip.freeze() outcome, whatever that is ... so when exactly should I normalize anything and most importantly, why is that, for a feature meant to freeze, indeed, dependencies?

hoodmane · 2024-10-25T16:44:53Z

Thanks for opening the issue @WebReflection. I'll copy my comments and we can discuss more over there.

WebReflection · 2024-10-28T10:00:13Z

This has been fixed and you should be able to test https://cdn.jsdelivr.net/npm/@pyscript/core@0.6.13/dist/core.js which is already on npm.

WebReflection · 2024-10-28T11:18:33Z

@Archmonger FYI we're out with latest release that fixes this and much more. You don't need to erase IndexedDB as the fix already takes care of previously stored packages by their own pinned version, it's just loading pyodide that changed so you should be covered.

On the other hand, remember that enforcing packages_cache = "never" in your config already erase entirely your IndexedDB so that in the future, or for developing/testing purpose, you should never have any issue with persistent IndexedDB (after all that's a persistent storage by definition but we can handle it with ease internally).

Archmonger · 2024-10-28T11:20:41Z

Thanks for the quick turnaround.

I'll bump the pyscript version in ReactPy shortly.

WebReflection · 2024-10-28T11:51:23Z

Thanks for the quick turnaround.

I'd rather thank you for the issue and the patience in explaining what's going on ... this also landed as manual test (due refresh needed across checks) but I think we've tackled the root of the issue and in the worst case scenario, our workaround will stick in time, in the best one, pyodide will handle this for us and we change a single line of code in the future trusting pyodide to handle this later on.

Also apologies we shipped something that clearly didn't have enough tests to guarantee all scenarios were covered, that's mostly on me as I never thought about pinned versions and belived that would've been handled out of the box.

Lesson learned from my side.

Archmonger · 2024-10-28T11:59:13Z

I haven't take a peek at your CI, but you could always have a live server test case using Python Playwright that handles the "manual refresh" nature of these tests. For example, we currently do some Playwright tests with "new tab" operations.

I am likely going to include refresh tests on my side of the fence moving forward, but could be good to have it your CI too.

WebReflection · 2024-10-28T12:55:03Z

The background thing is that I didn't think about pinned versions so that even implementing your new tab strategy would've failed your expectations but I need to dig into the new tab approach to validate twice everything is fine: that's a very good hint that might become handy in more than just this occasion.

I will open an issue about this, thanks for the hint!

WebReflection · 2024-10-28T13:00:09Z

Here we go, I'll try to tackle that sooner than later: #2233

Archmonger added needs-triage Issue needs triage type: bug Something isn't working labels Oct 23, 2024

Archmonger mentioned this issue Oct 23, 2024

Cleanup PyScript renderer & use Bun for building JS reactive-python/reactpy-django#254

Merged

4 tasks

WebReflection mentioned this issue Oct 25, 2024

lockFileURL fails at resolving from the underlying frozen object pyodide/pyodide#5135

Open

WebReflection added a commit to WebReflection/pyscript that referenced this issue Oct 28, 2024

Fix pyscript#2228 - Workaround in Polyscript for lockFileURL

b79f654

WebReflection added a commit to WebReflection/pyscript that referenced this issue Oct 28, 2024

Fix pyscript#2228 - Workaround in Polyscript for lockFileURL

fc85c9f

WebReflection mentioned this issue Oct 28, 2024

Fix #2220 - Delay plugins resolution due Safari 17.6 greedy resolution #2229

Merged

2 tasks

WebReflection closed this as completed in #2229 Oct 28, 2024

WebReflection closed this as completed in 9233d5e Oct 28, 2024

Archmonger mentioned this issue Feb 7, 2025

Configuring "packages" to a local wheel causes dependency resolution failure #2282

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V0.6.x micropip often fails to discover package #2228

V0.6.x micropip often fails to discover package #2228

Archmonger commented Oct 23, 2024 •

edited

Loading

WebReflection commented Oct 23, 2024

Archmonger commented Oct 23, 2024 •

edited

Loading

WebReflection commented Oct 24, 2024

WebReflection commented Oct 25, 2024 •

edited

Loading

WebReflection commented Oct 25, 2024

hoodmane commented Oct 25, 2024

WebReflection commented Oct 25, 2024 •

edited

Loading

hoodmane commented Oct 25, 2024

WebReflection commented Oct 28, 2024

WebReflection commented Oct 28, 2024 •

edited

Loading

Archmonger commented Oct 28, 2024

WebReflection commented Oct 28, 2024

Archmonger commented Oct 28, 2024

WebReflection commented Oct 28, 2024

WebReflection commented Oct 28, 2024

V0.6.x micropip often fails to discover package #2228

V0.6.x micropip often fails to discover package #2228

Comments

Archmonger commented Oct 23, 2024 • edited Loading

Checklist

What happened?

What browsers are you seeing the problem on? (if applicable)

Console info

Additional Context

WebReflection commented Oct 23, 2024

Archmonger commented Oct 23, 2024 • edited Loading

WebReflection commented Oct 24, 2024

WebReflection commented Oct 25, 2024 • edited Loading

WebReflection commented Oct 25, 2024

hoodmane commented Oct 25, 2024

WebReflection commented Oct 25, 2024 • edited Loading

hoodmane commented Oct 25, 2024

WebReflection commented Oct 28, 2024

WebReflection commented Oct 28, 2024 • edited Loading

Archmonger commented Oct 28, 2024

WebReflection commented Oct 28, 2024

Archmonger commented Oct 28, 2024

WebReflection commented Oct 28, 2024

WebReflection commented Oct 28, 2024

Archmonger commented Oct 23, 2024 •

edited

Loading

Archmonger commented Oct 23, 2024 •

edited

Loading

WebReflection commented Oct 25, 2024 •

edited

Loading

WebReflection commented Oct 25, 2024 •

edited

Loading

WebReflection commented Oct 28, 2024 •

edited

Loading