$300k grant for Python SDK for NEAR Protocol #5467

frol · 2025-01-06T11:45:03Z

frol
Jan 6, 2025

Hi RustPython community, I am a software engineer currently leading DevTools ecosystem in NEAR Protocol.

NEAR Protocol is a blockchain, that is resource-efficient, horizontally scalable, uses Wasm as the runtime, and is written in Rust. There are already Rust, JavaScript, and AssemblyScript SDKs to build apps. NEAR Protocol provides a few dozen host functions (Storage API, data input/output API, native cryptography API, etc), and has a limit of 1.5MB for Wasm file.

We have played a bit with RustPython and MircoPython. We could not make RustPython Wasm files fit the size requirement (<1MB, ideally, with a hard limit of 1.5MB). With MicroPython we succeeded in building a Wasm file. Still, it failed to be pre-compiled with the Wasm runtime in nearcore due to some unsupported Wasm features or functions that are not provided by NEAR host runtime.

The current error message we get from wasmer (Wasm runtime used in nearcore) with MicroPython

PrepareError(DeserializeError) means that there are some validation errors (e.g. imports that are not provided by Wasm host or unsupported Wasm features like bulk memory).

Milestone 1

I can offer $50k grant for the initial stage - get the following Python code to be compiled to Wasm (<1.2MB) and executed by Wasm runtime in nearcore without errors.

### NEAR API - START

def near_export(fn):
    """
    Decorator that exports the function as Wasm export function
    """
    # TODO

def near_input():
    """
    A wrapper for `input` + `read_register` host-functions provided by NEAR Protocol Wasm runtime.

    Ref:
    * `input`: https://gist.github.com/vgrichina/10ea303ab1bbf3d4069cd30d28f6340e#file-web4-min-zig-L9
    * `read_register`: https://gist.github.com/vgrichina/10ea303ab1bbf3d4069cd30d28f6340e#file-web4-min-zig-L24-L30
    """
    # TODO

def near_value_return(data_bytes):
    """
    A wrapper for `value_return` host-function provided by NEAR Protocol Wasm runtime.

    Ref: https://gist.github.com/vgrichina/10ea303ab1bbf3d4069cd30d28f6340e#file-web4-min-zig-L20-L22
    """
    # TODO

### NEAR API - END

import json

@near_export
def echo():
    input_bytes = near_input(0)
    near_value_return(json.dumps({"input_length": len(input_bytes)}))

Useful references:

Here is a minimal Zig implementation that can be compiled to NEAR-compatible Wasm: https://gist.github.com/vgrichina/10ea303ab1bbf3d4069cd30d28f6340e. Note, the exported symbols are the functions that NEAR can execute.
Here is a minimal Rust SDK with a minimal app compilable to NEAR-compatible Wasm: https://github.com/austinabell/nesdie/blob/main/examples/raw-contract/src/lib.rs
RustPython quick and dirty attempt that I got only down to 4.3MB: https://github.com/frol/near-contract-python-proof-of-concept
MicroPython attempt: micropython/micropython@master...miloserdow:micropython:master
I am happy to onboard you into the NEAR specifics

Milestone 2

I offer $50k grant to develop CLI tooling to streamline the build process that will allow multiple source files and PiPy dependencies (probably with the limitation that only Python-only dependencies will be suitable) to be packaged into a Wasm file that will work on NEAR (we still need to comply with 1.5MB filesize limit).

Take inspiration from cargo-near, which is CLI tool for building Rust apps for NEAR.

Consider using uv tooling.

Beyond

I offer up to $200k in grants for near-sdk-py development and improvements that will bring Python SDK on par with Rust SDK, providing data collections API over basic key-value Storage API, etc.

If you are interested in taking the challenge on, please, comment here or reach out to me in DM NEAR's Zulip @frol or Telegram.

lastmjs · 2025-01-09T04:35:52Z

lastmjs
Jan 9, 2025

Hello, Jordan from Demergent Labs. We might be able to take on this opportunity. We've worked extensively with RustPython to get our Python SDK for ICP (Internet Computer Protocol) to fit within its limits (which started out around ~2 MB gzipped but are fortunately much larger now).

What you're asking for will be a very difficult task depending on NEAR's protocol limitations. We've worked extensively with the teams behind ICP to improve the ICP protocol to allow for larger Wasm binary sizes and to enable more and more of the Python stdlib.

The Python stdlib in RustPython is very large. I would almost say it's impossible to fit it inside of 1.5 MB. The question becomes one of trade-offs and NEAR protocol improvements. For example, which Python stdlib functionality are you willing to live without? What is NEAR willing to do to improve the protocol to allow for larger binaries, Wasi, etc?

Another important point is Wasi. Does NEAR support the Wasi host imports? Those are very important for certain stdlib functionality.

ICP started out only supporting the wasm32-unknown-unknown Rust target and no Wasi functionality. Now Wasi is at least supported through a polyfill.

And so, please reach out to discuss this grant and future work with us. I am @lastmjs on Telegram and Twitter. I've sent you some messages on Telegram.

Our Python SDK for ICP: https://github.com/demergent-labs/kybra
Our TypeScript/JavaScript SDK for ICP: https://github.com/demergent-labs/azle
Our issues in RustPython: https://github.com/RustPython/RustPython/issues/created_by/lastmjs
Our PRs in RustPython
- add wasmbind feature #4875
- This is a draft to show our remaining reasons for having a fork of RustPython #4878

P.S. You're actually using our fork of RustPython in your demo

3 replies

frol Jan 9, 2025
Author

It was amazing to catch up with you in DM and just for the record of this discussion:

We got the minimal Rust contract using only near-sys going, the trick was not only compile it with cargo build --target wasm32-unknown-unknown --release, but also either use Rust 1.81 (newer versions pull in new Wasm features unsupported by Wasm VM in NEAR) or run the binary through wasm-opt --strip-debug release.wasm -o release-stripped.wasm, so the unsupported (and unused) Wasm features to be removed from the Wasm file.

I anticipate the following features to be needed for Python SDK for NEAR:

Python language (variables, functions, classes, imports, decorators, etc)
Python standard types: int, float, str, dict, set, list, bytes, array (view buffers)
Some Python standard modules: json, base64
Some external Python packages that do data encoding (like base58), hashing (though most popular hashing is already available via host functions), serialization/deserialization (most important one is borsh), and cryptography

Note: I believe that realistically we can only support pure-Python packages out there and maybe we can expose some ways to include Rust crates as Python modules via PyO3 - that will be size-efficient and runtime-efficient

NEAR does not provide WASI host functions, and I don't think we need those any time soon.

We also don't need to run numpy, sklearn, django, flask, sqlalchemy or anything like that as smart contracts, so this reduces the required set of features I believe.

petersalomonsen Jan 9, 2025

For the QuickJS runtime to run inside a near smart contract, QuickJS also requires WASI, but this is solved by implementing the WASI functions inside the wasm32-unknown-unknown binary. Functions such as wasi_clock_time_get and fd_write are implemented by calling similar functions on the smart contract environment imports. Not all WASI functions are used either, so then they are just mocked so that the imports are not added to the wasm binary. You can see how it is implemented here: https://github.com/petersalomonsen/quickjs-rust-near/blob/main/src/wasimock.rs

frol Jan 11, 2025
Author

Here is a repo + guide on how to add more logs to nearcore to troubleshoot errors that prevent or crash Wasm execution since by default nearcore won't provide any useful logs using near-workspaces-rs to test your Wasm file: https://github.com/frol/near-contract-minimal-test

youknowone · 2025-01-09T08:09:34Z

youknowone
Jan 9, 2025
Maintainer

Hi, I am currently on a trip, so the discussion can be a bit delayed.

I believe @lastmjs is skilled at developing blockchain SDK and have much trouble shooting experience with RustPython.

As one of the developer of RustPython, I am interested in milestone 1 and 2 to enhance the entire RustPython WASM ecosystem.

The Python stdlib in RustPython is very large. I would almost say it's impossible to fit it inside of 1.5 MB.

I agree here. there must be effort to cut-down the useless libraries. At the same time, I hope we also can reduce the wasting binaries to allow more third-party libraries.

0 replies

frol · 2025-03-13T21:04:16Z

frol
Mar 13, 2025
Author

This proposal is closed as Milestones 1 and 2 were achieved with MicroPython already: https://github.com/orgs/micropython/discussions/16427#discussioncomment-11986599, and Milestone 3 is built on top of it: https://github.com/r-near/near-sdk-py

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

$300k grant for Python SDK for NEAR Protocol #5467

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 3 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

$300k grant for Python SDK for NEAR Protocol #5467

frol Jan 6, 2025

Milestone 1

Milestone 2

Beyond

Replies: 3 comments · 3 replies

lastmjs Jan 9, 2025

frol Jan 9, 2025 Author

petersalomonsen Jan 9, 2025

frol Jan 11, 2025 Author

youknowone Jan 9, 2025 Maintainer

frol Mar 13, 2025 Author

frol
Jan 6, 2025

Replies: 3 comments 3 replies

lastmjs
Jan 9, 2025

frol Jan 9, 2025
Author

frol Jan 11, 2025
Author

youknowone
Jan 9, 2025
Maintainer

frol
Mar 13, 2025
Author