Skip to content

RFC: PyPI packaging guidelines #413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pfalcon opened this issue Apr 1, 2014 · 23 comments
Closed

RFC: PyPI packaging guidelines #413

pfalcon opened this issue Apr 1, 2014 · 23 comments
Labels
rfc Request for Comment

Comments

@pfalcon
Copy link
Contributor

pfalcon commented Apr 1, 2014

Let's discuss conventions and guidelines when publishing modules on PyPI - for "stdlib" (#405, https://github.com/micropython/micropython-lib), and in general.

Terminology: https://docs.python.org/3.3/distutils/introduction.html#distutils-specific-terminology

Proposal: if distribution wants to show strong affinity for being used with MicroPython (that's in particular 100% the case for micropython-lib), it's name should start with a standard prefix. Given our existing choices for long, fully explicit names, the prefix almost certainly should be "micropython-". That's long, but fully self-describing. So, last chance for a big crowd to shout "It's too long" and propose something shorter.

@dpgeorge
Copy link
Member

dpgeorge commented Apr 1, 2014

Aren't you the big proponent for short names? :)

After using it for some time, I agree that typing ./micropython is a bit long. Still, there is no better option, and having, eg, "upython-" as a prefix is not very descriptive.

@pfalcon
Copy link
Contributor Author

pfalcon commented Apr 1, 2014

Yeah, now that it's chosen, need to be consistent. Ok, here we go: https://pypi.python.org/pypi/micropython-unittest/0.0.1 released from https://github.com/micropython/micropython-lib/tree/master/unittest

Attempts to bend pip to install it: http://forum.micropython.org/viewtopic.php?f=3&t=55 (warning: thoughtstream-style).

@dhylands
Copy link
Contributor

dhylands commented Apr 1, 2014

After using it for some time, I agree that typing ./micropython is a bit long. Still, there is no better option, and having, eg, "upython-" as a prefix is not very descriptive.

Umm. You know about tab completion right?

./mTAB

@dholth
Copy link

dholth commented Apr 3, 2014

If you ask nicely you could probably get a Trove classifier for your implementation.

@dpgeorge
Copy link
Member

dpgeorge commented Apr 3, 2014

Umm. You know about tab completion right?

Yes! But sometimes I like to type the whole thing... :)

@pfalcon
Copy link
Contributor Author

pfalcon commented Apr 6, 2014

Following RFC #405 (comment) , I started to push patched CPython stdlib modules on PyPI. This includes first module with dependecy on another module, here's how to do it: https://github.com/micropython/micropython-lib/blob/master/copy/setup.py . Using dependencies requires setuptools; setuptools are not part of Python stdlib and should be installed separately (but of course it's usually preinstalled on any distro with decent Python support). I initially dropped initial attempts of using setuptools in favor of distutils exactly because distutils are in stdlib (and IMHO has better docs). But I wonder if we should just consistently use setuptools after all.

@pfalcon
Copy link
Contributor Author

pfalcon commented Apr 6, 2014

Example of installing from PyPI (assuming you have working pip on the host already):

$ cd tools/
$ ./pip-micropython install micropython-copy
...
$ ../unix/micropython
>>> import copy
>>> copy.copy({1:2})
{1: 2}

@pfalcon
Copy link
Contributor Author

pfalcon commented Apr 6, 2014

I went on and submitted all modules we currently have in micropython-lib (all of them are very rough). Current list is here: https://pypi.python.org/pypi?%3Aaction=search&term=micropython&submit=search

Please join the testing, discussion, and setting up guidelines and best practices for MicroPython module publishing!

pfalcon referenced this issue in micropython/micropython-lib Apr 7, 2014
@lurch
Copy link
Contributor

lurch commented Apr 9, 2014

I remember reading (a while ago) that setuptools was being deprecated...
So I've just done some googling and found there's lots of outdated info about python packaging (seems there were many false starts), but found that https://python-packaging-user-guide.readthedocs.org/en/latest/current.html is the current state of play, and it seems as though setuptools is again the (currently) recommended option.

@pfalcon
Copy link
Contributor Author

pfalcon commented Apr 12, 2014

FYI, having bleed-to-metalevel problem with packaging some standard modules: https://bitbucket.org/pypa/setuptools/issue/187/

@pfalcon
Copy link
Contributor Author

pfalcon commented Apr 12, 2014

The last issue was worked around in 06526e7, it's now possible to install micropython-re-pcre. Please help test.

@lurch
Copy link
Contributor

lurch commented Apr 14, 2014

Dunno if it's a problem or not, but I just noticed that if I tools/pip-micropython install micropython-re-pcre and then tools/pip-micropython install micropython-json, then installing micropython-json seems to re-install micropython-re-pcre as well?

@pfalcon
Copy link
Contributor Author

pfalcon commented Apr 14, 2014

FYI, having bleed-to-metalevel problem with packaging some standard modules: https://bitbucket.org/pypa/setuptools/issue/187/

Ok, so the underlying issue here was that there was no way to preclude putting a current (or more exactly, running script's) directory into sys.path. That's pretty annoying and serious issue, and I kinda even wanted to report it to CPython bugtracker. But now I was re-reading 3.4 changes, and saw:

The python command has a new option, -I, which causes it to run in “isolated mode”, which means that sys.path contains neither the script’s directory nor the user’s site-packages directory, and all PYTHON* environment variables are ignored (it implies both -s and -E).

So, that's it, but perhaps too much for a case in hand (only not forcing script dir to sys.path is needed for case at hand).

@pfalcon
Copy link
Contributor Author

pfalcon commented Apr 14, 2014

@lurch: Ack. Apparently, that's the effect of --root switch: pip thinks it installs stuff in virtualenv, and looks for existing packages there. But behind pip's back, we tell setuptools to put stuff under quite different path (--root), voila.

Btw, I had to bring in virtualenv exactly to make pip never consider any package as already installed on your system. Because otherwise it will refuse to install it again - in quite different location, where it actually doesn't exist. Actually, there's --ignore-installed option for pip, but it doesn't work as expected.

So, it's kinda impasse - we need that option to make packages robustly install, and that exactly causes them to always install from scratch. Wanna help with that? ;-). Warning: that's a trap for someone who wants to do stuff pedantically - the only outcome can be desire to rewrite it all from scratch. That's exactly what we can't/shouldn't do, so target should be the simplest and quickest solution (likely, workaround) satisfying requirements.

@lurch
Copy link
Contributor

lurch commented Apr 15, 2014

Afraid I've never looked into how pip works, and I've never even (knowingly) used a virtualenv.
But maybe I'll come back to it if I get bored and run out of files to 'make more pythonic' ;-)

@vitiral
Copy link
Contributor

vitiral commented Mar 25, 2015

I don't understand how "upy" is not descriptive. What else is called upy?

@LinJing517
Copy link

What would you do if you wanted to use SQLite3 on pyboard to indicate the absence of the "import ffi" module?

@dpgeorge
Copy link
Member

@PakchoiFood your comment doesn't make sense for this issue, did you post in error?

@dpgeorge
Copy link
Member

dpgeorge commented Sep 8, 2017

uPy-specific packages have been published using the micropython- prefix for the package name, and upip has been working well for quite some time on unix and (wifi-enabled) baremetal ports. So closing this issue.

@dpgeorge dpgeorge closed this as completed Sep 8, 2017
@pfalcon
Copy link
Contributor Author

pfalcon commented Sep 8, 2017

Yeah, but this ticket was supposed to be open, so people were able to find it and learn how to make a uPy-compatible package.

@dpgeorge
Copy link
Member

people were able to find it and learn how to make a uPy-compatible package.

IMO that's more suitable for documentation.

@pfalcon
Copy link
Contributor Author

pfalcon commented Sep 10, 2017

But we don't have such documentation, and this ticket is all we have.

@dpgeorge
Copy link
Member

But we don't have such documentation, and this ticket is all we have.

The README.md of the micropython-lib repo has some documentation, and points to this ticket for further reading. I don't see the need to keep this one open (it's still searchable and commentable if necessary). Feel free to take bits from here and add to the micropython-lib README.md if you think that's useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rfc Request for Comment
Projects
None yet
Development

No branches or pull requests

7 participants