Skip to content

[IDEA] Web imports for WASM #659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
palaviv opened this issue Mar 11, 2019 · 11 comments
Closed

[IDEA] Web imports for WASM #659

palaviv opened this issue Mar 11, 2019 · 11 comments

Comments

@palaviv
Copy link
Contributor

palaviv commented Mar 11, 2019

When using our WASM code currently doing import to anything that is not packaged in the build (builtins, stdlib) will not work. I suggest we will allow import in a way very similar to any other web resource.
For example when we will run the following from http://test.org/py.html

rp.pyEval("import test")

The WASM import will do a HTTP GET request to http://test.org/test.py and import the content of the file.

We can also allow the users to add other paths to sys.path so the browser will try to look for the imported file there as well. I think that the paths should be treated like html paths:

  • path/to/import: will be relative to the current file.
  • /path/to/import : will be absolute to the server root.
  • 'http://rustpypi.org/imports': will look in the imports path at domain rustpypi.org.

We should probably add the js VirtualMachine a setSysPath function like setStdout.

@palaviv palaviv mentioned this issue Mar 11, 2019
@coolreader18
Copy link
Member

Related to #228

@palaviv
Copy link
Contributor Author

palaviv commented Mar 11, 2019

Related to #228

I think that the solution at pyodide is a little more complex then what I suggest. I believe we should start from something simple like importing pure python code first.

@coolreader18
Copy link
Member

coolreader18 commented Mar 12, 2019

We could also use the PyPI API to fetch packages: https://warehouse.readthedocs.io/api-reference/json/; i.e. import x will look up x on PyPI and try to load it all somehow. The problem with both of these strategies is that import is synchronous, as is vm.exec(), while fetch and related APIs are async.

@palaviv
Copy link
Contributor Author

palaviv commented Mar 12, 2019

I think that we should start with synchronous fetch maybe by using XMLHttpRequest. After that I think we can try and do something more complex. We can send an asynchronous request when we encounter the import but wait for the import to finish only on the first use.

@coolreader18
Copy link
Member

coolreader18 commented Mar 12, 2019

What if we just had a WasmVM method/browser module export that accepted a module URL and returned a promise for when it was loaded? That way you would either load it from JS or do

def main():
    import module
browser.load_module("module", "https://module.url").then(main)

@palaviv
Copy link
Contributor Author

palaviv commented Mar 12, 2019

This will work great with new code someone writes specifically for RustPython WASM. I think that we would like a solution that will allow a user to use existing code in the regular way he used to.
I can't actually think about how to do what I suggest right now so I think starting with your suggestion and synchronous fetch would be best. We might want to add the pyEval a parameter with list of modules the user can ask to be loaded before the code is executed:

rp.pyEval("import module; module.cool_func()", {"imports": {"module": "https://module.url"}})

@coolreader18
Copy link
Member

I'm opposed to using synchronous fetch; it's effectively deprecated by web standards and having it be the main or only way to load Python modules would make for a horrible UX on sites that might use RustPython, as it freezes the entire page while it's loading.

@palaviv
Copy link
Contributor Author

palaviv commented Mar 12, 2019

I totally agree with your point except that I would not like to force a user to convert every code he has for it to work. I don't have a good option but maybe we should let the user decide how to do it. Once the option to override __import__ will exist the user can actually decide for himself. We should probably just provide the tools. I am in favor of adding a synchronizes function that the user can override __import__ with and have a big red warning that say it is a bad option.

@coolreader18
Copy link
Member

Alright, I guess that's fair.

@palaviv
Copy link
Contributor Author

palaviv commented May 5, 2019

Now that #914 is in I think the next steps are:

  • Allow relative paths
  • Search in sys.path as possible locations for module.
  • Support import chain (a.py import b.py).
  • Add import example to demo.

I think we might want to add some mechanism for "pre" imports that will allow defining a list of imports that will happen before the code is executed.

@coolreader18
Copy link
Member

Something I was thinking about was transforming the AST of a module that uses imports to be more like the example you gave for using browser.load_module, but I'm not sure if that would end up working.

@palaviv palaviv closed this as completed Sep 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants