[RFC] What are the compatibility goals for RustPython? #1940
Comments
|
I thought of another possibility related to the kernel Python blog post: use Rust's feature flags to define what parts of the stdlib are included. This would be really powerful for a couple reasons:
|
|
Hi @jamestwebber I will apologize in advance as I might not answer all of your questions. In addition this is my opinion and different people in the core team see the project going different ways. |
No problem, I definitely started rambling. Mostly I'm just curious. If the core team set down a roadmap for what they envision it would be useful for newcomers and also clarify your own priorities.
No GIL is very exciting! Imagining what might break gives me undergrad CS flashbacks but it'd be a big accomplishment. The big question for me is the C API. If RustPython can use e.g. the existing numpy package (either recompiled or even better the wheel) that would be an amazing benefit for the project and I think it would get a lot of people interested faster. There are tons of important Python+C packages out there and no one has the time to adapt them to a new interpreter. I think (might be naïve) that RustPython could be written to use them as-is. The C extension issue has been a hurdle for other alt-interpreters (like PyPy).
I was thinking about the parts which are still being written in Rust. Implementing the entire stdlib is going to be a long project, but a lot of people would be happy with a complete language implementation and pip. It would also be of interest to the broader community to show a demonstration of the "kernel Python" concept. As more pieces of the stdlib are ported to Rust, they could be written as feature flags. Someone interested in embedded RustPython might only want the kernel, whereas someone who wants broad compatibility will include everything you make available. Having a status page for these features would make it easy for a developer to check if their package works in RustPython: check the page to see if all the needed stdlib packages are implemented. There are some pieces (e.g. the stuff in PEP 594) that maybe you never implement, but having that listed on the status page as "needs volunteer" would make it clear and also somewhat validate that PEP (it's a bit contentious). Doug Hellmann has a useful post about what kernel python needs. Unsurprisingly there is a lot of cross-dependencies across the stdlib just to get pip working. Some of them are obvious ( If you wanted to go down this route, it makes sense to focus on parts of stdlib necessary for pip itself (some tweaks can make this list shorter), and also set up the status pages/documentation for the feature flag approach. |
Indeed the C API is a huge issue. One idea as detailed here #158 is to use hpy which is a better C API being developed to hide interpreter implementations internals from C extensions. The CPython C API is one of the main reasons why CPython can't get rid of the GIL and also why it can apply many optimizations that could make it much faster. The current C API exposes internals like memory layout of objects that current C Python extensions depend on, so that can't be changed to better implementations without breaking it. hpy hides all of that behind an opaque handle and provides functions as the only interface. Rust Python should probably adopt hpy. Mimicking the existing C Python API would be the wrong approach and a nightmare as you have to simulate the memory layout of objects, which is the problem PyPy has. |
|
HPy is actually a great target I think. HPy has their universal ABI target and will - eventually - be supported by Cython which a lot of extensions use, even Numpy. Currently not all the functions are implemented in Hpy and custom types are a work in progress. something to keep an eye on |
|
Ah those articles are great and expose my naïveté about Python internals. Clearly the existing C API makes a lot of things difficult. But luckily the necessary changes are already in progress on the CPython side and hopefully that will provide the incentive for major projects to adopt the better versions, which RustPython can take advantage of. In some ways it seems like some of the advantages of HPy are things that Rust makes simple (e.g. preventing borrowed references), so aiming for that interface makes a lot of sense, and it fits in with the "Python of the future" style. |
Couldn't agree more. As a user, knowing what I can and can't use without setting up the entire tool chain would be extremely useful. Even if the check boxes are very broad, some kind of checklist like the one gitoxide has would really help. I'm fine, even excited, by the deviation from CPython, but I'd like to see the rough differences instead of stabbing in the dark. |
|
as a pure python project, https://github.com/Pylons/waitress |
Uh oh!
There was an error while loading. Please reload this page.
Summary
As a long-time Pythonista and short-time Rustacean I of course searched for this project and was happy to find it existed. Looks like a lot of fun, and if I get the itch I'll definitely try to contribute (with the caveat that like most of you, I won't be able to dedicate a ton of time).
I was wondering about the long-term vision for RustPython in terms of CPython interop (if not necessarily parity). I think this question is the core of issues like #1849, #1815, #1690, #1204 and #1192, and having read those discussions it seems like it's an ongoing discussion.
Is RustPython a replacement for CPython, or a new interpreter to enable Python in new places (e.g. the web) but doesn't try to be compatible?
I think this is the big question, and answering it would define a roadmap for the future. The two answers define two different projects that are both exciting for different ways:
To me, RustPython is a cool idea in large part because Rust has good compatibility with C, and so RustPython could (maybe) work with the huge number of libraries that mix C with Python†. Most of the heavy hitters in the CPython space fall into this category–the whole data stack, for instance. I think e.g. PyPy suffers a lot from not being trivially compatible with such popular projects, even though it has lots of other advantages. Going this route means supporting all the complex meta-programming, introspection, etc that those projects rely on. That might be fun, if the goal is to learn about interpreters!
This approach doesn't mean not compiling to Wasm or anything like that. But if the goal is primarily web-native Python, I think that changes priorities and the direction of the project.
† not to mention Cython, C++, Fortran, etc: if it works with CPython then it could work with RustPython.
On the other hand, if the idea is to build a Python-to-wasm compiler/interpreter (or something like that), then CPython compatibility is a lot less important. This is more like Łukasz Langa 's proposal: you can discard weird code behavior and syntax, optimize how the language works (e.g. #1690, #1831), etc. That path is also super exciting to work on but is probably going to remain a toy project for a long time because you won't support very much outside of the stdlib. As Łukasz points out, a lot of defunct projects started out that way, and even PyPy is still a fairly niche project.
Drawbacks, Rationale, and Alternatives
There are pluses and minuses to both approaches. Both versions need to implement stdlib one way or another, so this decision doesn't immediately affect the direction of the project, but I think it's useful to discuss (and I was curious).
A somewhat "middle-road" answer would be to target compatibility with Python 3.10 or something, in the sense that the CPython roadmap defines some simplifications of the language. e.g. PEP 594 which removes some libraries from the stdlib (and you could be even more aggressive, as mentioned in #1204 and this post), and PEP 611 which defines some internal limits. I could imagine RustPython as a place that provides almost-complete compatibility with CPython but exposes more of the future, both in terms of new features and in terms of removing obsolete things.
Unresolved Questions
I don't think this is a decision that needs to be made, but having a general consensus among the core developers would help clarify what this project is for and give newcomers a sense of what to work on.
This turned into more of a screed than I intended, but I think it's a super interesting project. As someone more on the data science side of Python, I'm always yearning for a faster, cleaner core runtime that I could actually help contribute to. So I'm pretty firmly on the side of Option A, but that's just my perspective. Curious to hear your thoughts.
The text was updated successfully, but these errors were encountered: