Skip to content

Emscripten #888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
piranna opened this issue Oct 1, 2014 · 48 comments
Closed

Emscripten #888

piranna opened this issue Oct 1, 2014 · 48 comments

Comments

@piranna
Copy link

piranna commented Oct 1, 2014

Dumb question... has anyone tried to compile MicroPython to Javascript by using Emscripten? Aiming to be fully syntax compatible with Python3 but with a minimal memory footprint would be a good alternative to other Python-2-Javascript compilers... :-D The main problem would be to compile MicroPython with LLVM, but after that, it should be straighforward :-)

@blmorris
Copy link
Contributor

blmorris commented Oct 2, 2014

I don't know if this helps to answer you question, but the unix port does now compile under clang, which as I understand is derived from LLVM. So if the aim is to compile under LLVM, then maybe the adaptations for clang could be applied?

@piranna
Copy link
Author

piranna commented Oct 2, 2014

If there's LLVM support, then it's possible to use emscripten :-) Remaining is the runtime files, but this then it's just a standard porting. Emscripten does,t oficially support Node.js, but it could ve a good starting point for testing and development...

@turbinenreiter
Copy link
Contributor

I tried and failed, but I think it is possible and a good idea. pypyjs already does the same thing pretty well, but the size of the js-file the browser has to load is to big. micropython may be able to fix that.
First, using emmake results in an error, claimng it can't find libffi. Don't know why.
To bypass that you can manually go into the Makefile and set CC = emcc. If you then make, it starts running, but then aborts once it comes to the assembly files, something like 'unknown file extions, *.S not supported'.

@piranna
Copy link
Author

piranna commented Oct 2, 2014

To bypass that you can manually go into the Makefile and set CC = emcc.
If you then make, it starts running, but then aborts once it comes to the
assembly files, something like 'unknown file extions, *.S not supported'.

I think it makes sense, both the browser and Javascript can't be able to
manage assembly code... ;-) It will need to be wrapped and exec the
assembler instructions "in software".

@turbinenreiter
Copy link
Contributor

I meant the compilation starts and aborts, saying: it seems like emscripten can't compile assembly to js.
I'd start a new port, starting from the unix port and replace all assembly with C.

Basically, this file:
https://github.com/micropython/micropython/blob/master/py/nlrx64.S

@piranna
Copy link
Author

piranna commented Oct 2, 2014

I meant the compilation starts and aborts, saying: it seems like
emscripten can't compile assembly to js.
I'd start a new port, starting from the unix port and replace all
assembly with C.

Yes, I think that's the way to go, too :-) Maybe the unix port could get
some porting beneficts from this! :-D

@pfalcon
Copy link
Contributor

pfalcon commented Oct 2, 2014

To bypass that you can manually go into the Makefile and set CC = emcc.

No, to bypass that, you disable building with libffi, which is of course of no use for javascript.

Maybe the unix port could get some porting beneficts from this! :-D

It can't, because it already has that. Quick look at *.S files will reveal the needed config settings.

@piranna
Copy link
Author

piranna commented Oct 2, 2014

libffi could be useful on Node.js, it could be used node-ffi, that has libffi bundled inside it.

@pfalcon
Copy link
Contributor

pfalcon commented Oct 2, 2014

What node.js has to do with micropython? Also, please consider moving all this topic to forum - this ticket doesn't represent any issue with micropython and we already have too many bugs open.

@turbinenreiter
Copy link
Contributor

close this, let's talk here:
http://forum.micropython.org/viewtopic.php?f=5&t=322

@dpgeorge
Copy link
Member

dpgeorge commented Oct 2, 2014

It would be nice to have uPy running in the browser. Main difficulties: GC (just disable it!) and exceptions (setjmp/longjmp).

@alanjds
Copy link

alanjds commented Mar 10, 2016

Hi. Any updates on compiling micropython to javascript?

@dhylands
Copy link
Contributor

Not from me - I've been swamped with other things.

@alanjds
Copy link

alanjds commented Mar 11, 2016

@dhylands I am using this line from your makefile to try compiling: make CC='emcc' CPP='gcc -E' CFLAGS_EXTRA=-DMICROPY_NLR_SETJMP MICROPY_PY_FFI=0 STRIP=echo SIZE=echo V=1

Is that the error you faced at that time?

/Users/alanjds/Downloads/emsdk_portable/clang/e1.29.0_64bit/llvm-link: link error in '/Users/alanjds/.emscripten_cache/libc.bc': Linking globals named 'snprintf': symbol multiply defined!

Not intended to waste other people time, just to know how near of the same "edge" I am right now.

@dhylands
Copy link
Contributor

IIRC I got it to compile and link, but it didn't run, but that was a few months ago and things are hazy now.

@alanjds
Copy link

alanjds commented Jul 14, 2016

Hi.

I want to put a bounty on this, but then the issue needs to be kept open.

Is this desirable? Can someone reopen the issue please?

@alanjds
Copy link

alanjds commented Jul 19, 2016

Should I open a new issue just for that?

@piranna
Copy link
Author

piranna commented Jul 19, 2016

It should be enough to re-open it, but maybe open a new one would be a good idea too...

@pfalcon pfalcon reopened this Jul 19, 2016
@pfalcon
Copy link
Contributor

pfalcon commented Jul 19, 2016

@alanjds : Sorry, that request was lost in the noise. Reopened now.

@alanjds
Copy link

alanjds commented Jul 21, 2016

Just issued a $100 USD bounty on this feature request issue.
Link: https://www.bountysource.com/issues/5037165-emscripten

The bounty is valid for the next 6 months, then expires.

Thanks @piranna for opening and @pfalcon for reopening this issue.

@pfalcon
Copy link
Contributor

pfalcon commented Jul 23, 2016

That's actually what I wanted to ask @alanjds : Thanks for starting this, this may be an important way to support MicroPython development going forward. But please specify what exactly needs to be implemented for someone to claim this bounty.

@alanjds
Copy link

alanjds commented Jul 23, 2016

Hi. I am glad that you liked and that someone (@ramsundhar20) already started to work on it!

I am not strictly sure and am open to changes, but the exactly needs to close this ticket (#888) and claim the bounty IMHO is:

  • a working port of micropython to the Javascript "architecture", possible via Emscripten or similar.
  • after ported, it needs to run in a standard desktop browser with Javascript enabled, although is desirable to run in a standard Android browser.
  • a REPL of micropython.js running in this browser
  • the ported result should pass the tests that other architectures passes, saved the environment-incompatible stuff (there is no direct socket access from browsers right now, I guess)

Just get into http://pypyjs.org/ and try to use the inline REPL. This is a PyPy interpreter compiled to asm.js via Emscripten. Then go http://pypyjs.org/editor.html and click "Run". It works. Edit with something compatible with standard PyPy and click "Run" again. It works again.

For me, the goal of this ticket here (#888) is to have a similar thing with micropython: a micropython.js

Go to http://arewepythonyet.com/. There is a bunch of benchmark comparison graphs there. Ideally, micropython.js should be able to be added to this graphs. Well, micropython does not have JIT, so no JIT for micropython.js, but what micropython does micropython.js is expected to do, right?

My personal final goal (aka "Evil Plan") is to run python-pure desktop stuff on an Android browser, with support for network access and filesystem abstractions, but this ticket goes not that far ahead (unless the developer got inspired ;).

For me, what PyPy.js had done is impressive and is a huge step towards the future. However, is kind of prohibitive to have ~3MB downloaded on first use, or ~10MB unpacked in memory on every mobile tab that uses it.

The real goal of this ticket is to have what PyPy.js offer, but for a very lower memory/network cost. Right now, CPU performance of micropython.js is not an issue, as far as is not for the standard micropython.

And thanks again @ramsundhar20 for stepping up.

@piranna
Copy link
Author

piranna commented Jul 25, 2016

As the one that openned the issue, I think I have something to say regarding what the bounty should be about ;-) I did not create it to make it possible to run micropython on the browser, but on Node.js, that by the way would be an easier to achieve task since Node.js environment is itself inspired on Python. Said so, and based on the list provided by @alanjds, I think this bounty can be clossed when:

  1. a working port of micropython to the Javascript "architecture" is done, possible via Emscripten or Cheerp or similar.
  2. the port needs to run on Node.js and provide an executable to run regular Python scripts.
  3. the executable should provide all the regular options of the CPython binary, and also an equivalent CPython REPL, all of this running on the Node.js environment.
  4. published as npm package

Bonuses:

  1. the ported result should pass the tests that other architectures passes, specially the ones regarding to pure Python ones
  2. allow to use Node.js modules as Python imports in a transparent and interoperable way (no wrapper functions)
  3. (optional) provide an instance of the Python Standard Library, ideally as a npm dependency after being ported to Javascript as an independent module importable and usable by other projects not related to the micropython port or with Python at all

@deshipu
Copy link
Contributor

deshipu commented Jul 25, 2016

@piranna sorry for asking this, it's possibly off-topic here, but I can't help my curiosity. What is the practical use of MicroPython running on Node.js? What is your use case?

Wouldn't you think that the person who offered the bounty should get to decide what they are paying for?

@alanjds
Copy link

alanjds commented Jul 25, 2016

Thoughts:

  1. This "Cheerp" thing looks nice! In-browser debugging, commercial support, direct JS calling access...
  2. Maybe is indeed better to split this in two or three tickets, money-backed:
  • a) JS backend (zero $)
  • b) Run on node.js ($ ??)
  • c) Run on browser ($ 100)

I vote for any bounty to be redeemed only if a+b or a+c got covered.

Not wanting to be rude, but what I really need is browser support. However, solving c) makes b) a lower hanging fruit, right?

@alanjds
Copy link

alanjds commented Jul 25, 2016

@deshipu: I see a node.js port as a way to use V8 JIT to speedup non-browser Python, but only for pure-python.

@piranna
Copy link
Author

piranna commented Jul 25, 2016

What is the practical use of MicroPython running on Node.js? What is your use case?

Use it on NodeOS, both as a Python interpreter, but also to be used with node-gyp, since GYP is build in Python (although they are fixing this need by using gyp.js instead).

However, solving c) makes b) a lower hanging fruit, right?

Well, solving the Node.js support, you can use browserify after that...

@deshipu: I see a node.js port as a way to use V8 JIT to speedup non-browser Python, but only for pure-python.

This would also happen on the browser, except if the Node.js port uses low-level v8 optimizations (that I think are offtopic...).

@piranna
Copy link
Author

piranna commented Jul 25, 2016 via email

@alanjds
Copy link

alanjds commented Jul 25, 2016

As a reference, this is how PyPy.js has done: https://youtu.be/8C9q94F6Uqo?t=13m18s

I am trying to gather some graph showing that the most important thing on PyPy.js is the compatibility, but had not found the presentation yet. Will post here as soon as I find it.

In this, Brython is compared too, as being very fast, but very "not compatible" (UPDATE: Brython is compared, but not the way I remembered...)

@alanjds
Copy link

alanjds commented Jul 25, 2016

Found!

@alanjds
Copy link

alanjds commented Jul 25, 2016

Just found a working CPython port to Javascript (emscripten-based): https://github.com/aidanhs/empython

@jumasheff
Copy link

One of the possible use cases: use micropython.js-based emulator for online
classes.

Currently I'm playing with Skulpt http://www.skulpt.org/ -- "entirely
in-browser implementation of Python" -- possible solution? I mean, why
convert micropython, when you already have Skulpt? Micropython specific
libs can be converted separately, see docs
http://www.skulpt.org/static/ProgMan/index.html.


Best regards / Урматым менен / С уважением,
Murat Jumashev / Мурат Жумашев

On Tue, Jul 26, 2016 at 4:21 AM, Alan Justino da Silva <
notifications@github.com> wrote:

Just found a working CPython port to Javascript (emscripten-based):
https://github.com/aidanhs/empython


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#888 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADKon9LFiURQACHW519UN5al0gLreimNks5qZTbSgaJpZM4Cpwj2
.

@alanjds
Copy link

alanjds commented Jul 26, 2016

@jumasheff I already evaluated Skulpt. The compatibility is not that great IMHO. Not everything you import in CPython works on Skulpt. But everything you import on PyPy works on PyPy.js ;)

@piranna
Copy link
Author

piranna commented Jul 26, 2016

@jumasheff I already evaluated Skulpt. The compatibility is not that great IMHO. Not everything you import in CPython works on Skulpt. But everything you import on PyPy works on PyPy.js ;)

Yes, while AFAIK PyPy is only Python 2 compatible, while MicroPython is focused on Python 3.4 :-)

@pfalcon
Copy link
Contributor

pfalcon commented Jul 26, 2016

@alanjds: Thanks for detailed description, I'm sure it provides enough insight for someone to start working on the task.

Everyone: place for general discussion, offtopic to this ticket is http://forum.micropython.org . Thanks.

@alanjds
Copy link

alanjds commented Aug 10, 2016

Looks like someone have a compiling version: #1561

Working is another story...

@alanjds
Copy link

alanjds commented Oct 13, 2016

Just to inform: Some weeks ago I contacted @ramsundhar20 and he will not continue seeking this ticket, by personal reasons.

So, it is available to be taken, if someone else does want to.

tannewt added a commit to tannewt/circuitpython that referenced this issue Jun 1, 2018
atmel-samd/samd51: Refactor clock setup + implement samd.clock
@jedie
Copy link
Contributor

jedie commented Sep 18, 2018

Isn't https://github.com/micropython/micropython-unicorn exactly what we want?

@piranna
Copy link
Author

piranna commented Sep 18, 2018

Isn't https://github.com/micropython/micropython-unicorn exactly what we want?

Sounds interesting, but that's micropython running in an emulated microcontroller written in Javascript, not directly in the Javascript VM. Compiling micropython to WebAssembly is more alike what we want here.

@alanjds
Copy link

alanjds commented Oct 1, 2018

@jedie I would prefer #3575. One less VM level and easier access to Javascript in the future, I guess.

@pmp-p
Copy link
Contributor

pmp-p commented Dec 23, 2018

@piranna, https://github.com/matthewelse/micropython/tree/emscripten/emscripten is already enough to get a working libmicropython.wasm, and with some work on main it it is easy to run any file http://pmpp.pagesperso-orange.fr/upy.js/

if anyone is still interested i have socket / time / dom access to canvas 2D are possible ( already made it with cpython ).

@kkimdev
Copy link

kkimdev commented Dec 23, 2018

@pmp-p There were other attempts in the past #3474 and making the garbage collector work correctly was the issue. Is there any progress on that?

@piranna
Copy link
Author

piranna commented Dec 23, 2018

@piranna, https://github.com/matthewelse/micropython/tree/emscripten/emscripten is already enough to get a working libmicropython.wasm, and with some work on main it it is easy to run any file http://pmpp.pagesperso-orange.fr/upy.js/

That looks cool, has you published it to npm?

@pmp-p
Copy link
Contributor

pmp-p commented Dec 23, 2018

@kkimdev garbage collection has already problem on its own, no need to mess with it until it's ok, also a webbrowser is not a calculator you have 512MiB free until heap overflow.

@piranna i'm not familiar with node ecosystem, but isn't npm for javascript libraries ? The libmicropython.wasm is a prebuilt made from C llvm bytecode.

@piranna
Copy link
Author

piranna commented Dec 23, 2018

Node.js support WebAssembly and tere's already a lot of wasm modules already there, most of them transpiled from Rust, so there's no problem with that.

@pmp-p
Copy link
Contributor

pmp-p commented Jan 30, 2019

Please test/help/pr here https://github.com/pmp-p/micropython-ports-wasm
i'm mostly interested in GC testing ( adding/testing del method) and C/C++/FFI dynamic linking with non wasm-lib ( AMS.js )

@dpgeorge
Copy link
Member

The port to javascript in #3575 was merged in 7d675f3

@dpgeorge
Copy link
Member

dpgeorge commented Jun 9, 2020

Closing because the port to Javascript exists.

@dpgeorge dpgeorge closed this as completed Jun 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests