Add a way to inject modules to rustpython_wasm #2402

coolreader18 · 2021-01-19T20:02:09Z

To add your own modules, you should make your own crate that depends on

# for wasm-bindgen
[lib]
crate-type = ["cdylib", "rlib"]

# can remove `branch="..."` once  this PR is merged
rustpython_wasm = { git = "https://github.com/RustPython/RustPython", branch = "wasm-hooks" }
rustpython_vm = { git = "https://github.com/RustPython/RustPython", branch = "wasm-hooks" }
wasm-bindgen = "0.2"

# might also want this so that compilation doesn't take forever
[package.metadata.wasm-pack.profile.release]
wasm-opt = false

Then your lib.rs should look like:

pub use rustpython_wasm::exports::*;

use wasm_bindgen::prelude::*;
use rustpython_vm::VirtualMachine;

#[wasm_bindgen(start)]
fn init() {
    rustpython_wasm::add_init_func(init_vm);
}

fn init_vm(vm: &mut VirtualMachine) {
    vm.add_frozen(rustpython_vm::py_freeze!(dir = "deps"));
}

deps/ should be a directory next to your Cargo.toml, with a subdirectory deps/pytezos with stuff like deps/pytezos/__init__.py.

satyajeetkolhapure · 2021-01-20T14:25:50Z

Thanks @coolreader18 for the quick fix.

I created new crate and cargo.toml looks like -

[package]
name = "PytezosRustPython"
version = "0.1.0"
authors = ["satyajeetkolhapure"]
edition = "2018"

[lib]
crate-type = ["cdylib", "rlib"]

[dependencies]
rustpython_wasm = { git = "https://github.com/RustPython/RustPython", branch = "wasm-hooks" }
rustpython-vm = { git = "https://github.com/RustPython/RustPython", branch = "wasm-hooks" }
wasm-bindgen = "0.2"

[package.metadata.wasm-pack.profile.release]
wasm-opt = false

And lib.rs looks like

pub use rustpython_wasm::exports::*;

use wasm_bindgen::prelude::*;
use rustpython-vm::VirtualMachine;

#[wasm_bindgen(start)]
fn init() {
    rustpython_wasm::add_init_func(init_vm);
}

fn init_vm(vm: &mut VirtualMachine) {
    vm.add_frozen(rustpython-vm::py_freeze!(dir = "deps"));
}

I have added simple mymodule.py file in deps folder which has one simple function

When I try to run wasm-pack build --target nodejs I get below error

Compiling rustpython-pylib v0.1.0 (https://github.com/RustPython/RustPython?branch=wasm-hooks#304a47af)
error[E0599]: no method named `into_pyexception` found for struct `std::io::Error` in the current scope
   --> /Users/satyajeetkolhapure/.cargo/git/checkouts/rustpython-f8ef4d934ac33cd8/304a47a/vm/src/stdlib/thread.rs:246:28
    |
246 |         .map_err(|err| err.into_pyexception(vm))
    |                            ^^^^^^^^^^^^^^^^ method not found in `std::io::Error`
    |
    = help: items from traits can only be used if the trait is implemented and in scope
note: `IntoPyException` defines an item `into_pyexception`, perhaps you need to implement it
   --> /Users/satyajeetkolhapure/.cargo/git/checkouts/rustpython-f8ef4d934ac33cd8/304a47a/vm/src/exceptions.rs:42:1
    |
42  | pub trait IntoPyException {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0599`.
error: could not compile `rustpython-vm`

Any idea, if I am missing something?

coolreader18 · 2021-01-20T14:48:26Z

Oh whoops, yeah, add default-features = false to the rustpython-vm dependency

satyajeetkolhapure · 2021-01-20T15:58:22Z

Sorry @coolreader18 for too many questions. This time cargo build succeeds, but when I ran wasm-pack build --target nodejs I got below error

[INFO]: ⬇️  Installing wasm-bindgen...
error: cannot specify two `start` functions
Error: Running the wasm-bindgen CLI
Caused by: failed to execute `wasm-bindgen`: exited with exit code: 1

May be because rustpython_wasm package already has start method. Not sure how to fix this.

coolreader18 · 2021-01-20T16:03:18Z

Oh, hmmmm. I didn't think that the one in rustpython_wasm would count since it isn't really exported.

coolreader18 · 2021-01-20T18:57:00Z

@satyajeetkolhapure try adding features = ["no-start-func"] to your rustpython_wasm dependency

satyajeetkolhapure · 2021-01-21T15:44:58Z

Hi @coolreader18

I managed to run new package with new changes. I am not sure how to use it though.

I created mymodule.py inside deps folder in my new crate. Created pkg using wasm-pack command.

mymodule.py looks like below

def getPrintString():
    return 'Hello world'

Then I created main.py like below

from mymodule import getPrintString
print(getPrintString())

I then created runwasm.js like below

const rp = require('/Users/satyajeetkolhapure/TruffleProject/TezosRustPython/PytezosRustPython/pkg_old/PytezosRustPython.js')
const fs = require('fs')

const content = fs.readFileSync('main.py').toString()
rp.pyEval(content)

But I am getting below error

Error parsing Python code: Got unexpected token 'print' at line 2 column 1
print(getPrintString())

Any idea?

coolreader18 · 2021-01-21T16:39:16Z

You should call pyExec, not pyEval, pyEval expects one expression. Also, you have to have mymodule.py in the deps/ folder that gets frozen, wasm doesn't have access to the file system.

satyajeetkolhapure · 2021-01-22T12:55:24Z

@coolreader18 : You are awesome. It worked. Thank you so much for all your help and solving this problem. I will keep you posted when I try it with pytezos library.

satyajeetkolhapure

I tested it. It works.

satyajeetkolhapure · 2021-01-23T12:18:56Z

Hi @coolreader18

While testing it for pytezos, I am getting error because of other dependencies which pytezos needs. Is it possible to copy whole venv as well with this? When I copy the whole venv folder inside deps folder, somehow it does not find packages.

e.g I have venv folder with all site-packages and one python file which uses those dependencies. How can I achieve it? Is there any way to add venv?

coolreader18 · 2021-01-23T22:35:28Z

@satyajeetkolhapure you should be able to point the py_freeze!(dir = "...") to your venv packages folder, though keep in mind that the more you include the larger the wasm binary will be

satyajeetkolhapure · 2021-01-25T14:34:45Z

@coolreader18 Somehow when I add packages folder with py_freeze, it is not able to detect it.

I did below steps -

Created new python project with venv python 3.7 version
Installed pytezos package
Copied all the folders and files from vevn/lib/python3.7/site-packages folder to deps folder inside my new crate rust project
Built pkg using wasm command
Tried to import pytezos, it failed with module not found.
Probably site-packages are treated differently. If I manage to add pytezos as std-library in vm as others like html, email etc. I might be able to use it.

Is it possible to add it as standard library. I mean if I copy all pytezos libraries inside pylib-crate/Lib folder on my local machine vm repository, will those be added as standard libraries? or do I need something more than just copying?

coolreader18 · 2021-01-26T01:50:11Z

@satyajeetkolhapure that's really weird, you can definitely try copying everything in deps/ to rustpython/Lib. Could you post what the contents of your deps/ folder looks like? Like just ls deps or something?

satyajeetkolhapure · 2021-01-26T12:12:59Z

@coolreader18 : Below are the folders and files inside deps folder

pycache
_distutils_hack
base58-1.0.3.dist-info
base58.py
certifi
certifi-2020.12.5.dist-info
chardet
chardet-4.0.0.dist-info
dateutil
distutils-precedence.pth
fire
fire-0.4.0-py3.7.egg-info
idna
idna-2.10.dist-info
loguru
loguru-0.5.3.dist-info
mnemonic
mnemonic-0.19.dist-info
netstruct-1.1.2-py3.7.egg-info
netstruct.py
pendulum
pendulum-2.1.2.dist-info
pip
pip-21.0.dist-info
pkg_resources
ply
ply-3.11.dist-info
pyblake2-1.1.2-py3.7.egg-info
pyblake2.cpython-37m-darwin.so
pytezos
pytezos-2.2.8.dist-info
python_dateutil-2.8.1.dist-info
pytzdata
pytzdata-2020.1.dist-info
requests
requests-2.25.1.dist-info
setuptools
setuptools-52.0.0.dist-info
simplejson
simplejson-3.17.2.dist-info
six-1.15.0.dist-info
six.py
termcolor-1.1.0-py3.7.egg-info
termcolor.py
testmain.py
tqdm
tqdm-4.56.0.dist-info
urllib3
urllib3-1.26.2.dist-info

I will give it a try copying all these inside Lib folder of rustpython

satyajeetkolhapure · 2021-01-26T15:17:15Z

Hi @coolreader18 : I am using python3.7 to create those packages. Probably, this is causing it, may be because of obsolete import rather than relative.

Which version of python interpreter RustPython uses?

coolreader18 · 2021-01-26T15:41:49Z

We don't really have a specific version we target, we're a mishmash of anywhere from 3.5 to 3.9. Have you changed your lib.rs at all? It's still vm.add_frozen(py_freeze!(dir = "deps")) ?

satyajeetkolhapure · 2021-01-26T15:52:55Z

yes. it is still the same vm.add_frozen(py_freeze!(dir = "deps"))

I tried even copying all those files in Libs folder in vm. Somehow it found the main module. But it failed to find submodule. the paths used are obsolute. Not sure if that is the reason.

coolreader18 · 2021-01-26T16:01:13Z

Which paths? (assuming you mean absolute). Like the imports are import pytezos.bar.baz instead of import ..bar.baz?

coolreader18 · 2021-01-26T16:01:47Z

There might be some unresolved issues with recursive relative imports, I know there are a few instances in the standard library we've had to work around.

satyajeetkolhapure · 2021-01-26T16:20:11Z

correct import pytezos.rpc is throwing error No module named 'pytezos.rpc'

could you help me with the workaround? if possible. if you point me to any example of workaround

coolreader18 · 2021-01-26T18:46:29Z

Were you ever able to get it working in the first place? Trying to import pytezos just using the native rustpython binary seems to error in a bunch of different ways.

satyajeetkolhapure · 2021-01-26T18:59:32Z

When I create new python project and installed pytezos package, I called pytezos as below

import pytezos from pytezos
print(pytezos)

Then I ran it in pycharm. I also tried command python3 filename.py

It gave me output

<pytezos.client.PyTezosClient object at 0x103cf59d0>

Properties
.key  # tz1grSQDByRpnVs7sPtaprNZRp531ZKz6Jmm
.shell  # https://rpc.tzkt.io/delphinet/ (delphinet)

Helpers
.account()
.activate_account()
.balance()
.ballot()
.contract()
.delegation()
.double_baking_evidence()
.double_endorsement_evidence()
.endorsement()
.nft_app()
.now()
.operation()
.operation_group()
.origination()
.proposals()
.reveal()
.seed_nonce_revelation()
.transaction()
.using()

Process finished with exit code 0

I had to install below two on mac os

$ brew tap cuber/homebrew-libsecp256k1
$ brew install libsodium libsecp256k1 gmp

pytezos documentation link - https://pypi.org/project/pytezos/

Appreciate your help @coolreader18

coolreader18 · 2021-01-26T19:01:16Z

Right, but were you ever able to get it working in rustpython? RustPython isn't fully compatible with CPython yet, and looking at what pytezos depends on, it looks like there are some native modules, which RustPython probably won't support any time soon.

satyajeetkolhapure · 2021-01-26T19:04:50Z

I was not able to run it using rustpython yet.

Possible. I think, I need to wait for those native modules be part of rustpython then.

Will it be a long task to add those native modules? Just to get an idea

coolreader18 · 2021-01-26T19:09:36Z

Well it wouldn't really be adding them, it would be adding support for native extensions. Alternatively, you could translate every native module that pytezos depends on to rust and include it in your wasm binary, but I don't even know if that's possible - it looks like there's lots of crypto stuff as well as http requests, which can't be made from wasm

coolreader18 · 2021-01-26T19:10:20Z

Sorry for "leading you on" for so long, I hadn't ever looked up what pytezos is until today, I don't think it's possible to run in rustpython at the moment

satyajeetkolhapure · 2021-01-27T11:21:35Z

@coolreader18 no problem mate

Add a way to inject modules to rustpython_wasm

304a47a

satyajeetkolhapure approved these changes Jan 20, 2021

View reviewed changes

Add no-start-func feature

be1bacd

satyajeetkolhapure approved these changes Jan 22, 2021

View reviewed changes

youknowone approved these changes Feb 3, 2021

View reviewed changes

coolreader18 merged commit f49448a into master Feb 3, 2021

youknowone deleted the wasm-hooks branch February 4, 2021 06:14

Add a way to inject modules to rustpython_wasm #2402

Add a way to inject modules to rustpython_wasm #2402

Uh oh!

Conversation

coolreader18 commented Jan 19, 2021

Uh oh!

satyajeetkolhapure commented Jan 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coolreader18 commented Jan 20, 2021

Uh oh!

satyajeetkolhapure commented Jan 20, 2021

Uh oh!

coolreader18 commented Jan 20, 2021

Uh oh!

coolreader18 commented Jan 20, 2021

Uh oh!

satyajeetkolhapure commented Jan 21, 2021

Uh oh!

coolreader18 commented Jan 21, 2021

Uh oh!

satyajeetkolhapure commented Jan 22, 2021

Uh oh!

satyajeetkolhapure left a comment

Choose a reason for hiding this comment

Uh oh!

satyajeetkolhapure commented Jan 23, 2021

Uh oh!

coolreader18 commented Jan 23, 2021

Uh oh!

satyajeetkolhapure commented Jan 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coolreader18 commented Jan 26, 2021

Uh oh!

satyajeetkolhapure commented Jan 26, 2021

Uh oh!

satyajeetkolhapure commented Jan 26, 2021

Uh oh!

coolreader18 commented Jan 26, 2021

Uh oh!

satyajeetkolhapure commented Jan 26, 2021

Uh oh!

coolreader18 commented Jan 26, 2021

Uh oh!

coolreader18 commented Jan 26, 2021

Uh oh!

satyajeetkolhapure commented Jan 26, 2021

Uh oh!

coolreader18 commented Jan 26, 2021

Uh oh!

satyajeetkolhapure commented Jan 26, 2021

Uh oh!

coolreader18 commented Jan 26, 2021

Uh oh!

satyajeetkolhapure commented Jan 26, 2021

Uh oh!

coolreader18 commented Jan 26, 2021

Uh oh!

coolreader18 commented Jan 26, 2021

Uh oh!

satyajeetkolhapure commented Jan 27, 2021

Uh oh!

Uh oh!

satyajeetkolhapure commented Jan 20, 2021 •

edited

Loading

satyajeetkolhapure commented Jan 25, 2021 •

edited

Loading