Skip to content

Extending the python standard libraries #281

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
royemmerich opened this issue Feb 13, 2014 · 3 comments
Closed

Extending the python standard libraries #281

royemmerich opened this issue Feb 13, 2014 · 3 comments
Labels
rfc Request for Comment

Comments

@royemmerich
Copy link

Disclaimer: I am on a steep learning curve. Take what I say with a pinch of salt and please correct gently ;)

Following on from my post in issue #83 about wanting to try out MODBUS RTU communication, I have spent the last week and a half investigating the various issues involved as well as trying to get my head around the micropython project structure.

Libraries are essential

As I see it, one of the biggest limitations to writing more complex applications is the lack of some rather useful python standard libraries as listed here:

http://docs.python.org/3/library/

Divide and conquer

From the sheer number of standard libraries available in CPython, it is going to take quite a lot of effort to support even a fraction of that functionality in micropython. It certainly isn't something @dpgeorge could or would want to tackle on his own.

Looking at the list it seems like a good idea to break it down into categories so that we can easily see what has been covered and what needs attention:

  • applicable
    • essential (focus on these first)
    • nice to have (then tackle these)
  • debatable (only for the curious)
  • not applicable
Which way Sam Gamgee?

Being the C/microcontroller noob around here (I cut my programming teeth on Java) but an engineer at heart, I like to have a clear path to follow to avoid chaos later on. I would really appreciate it if @dpgeorge and the rest of you experts could thrash out a bit of a framework for the rest of us to follow. Just some quick dos and don'ts would be good and maybe a bit of input about the overall design so we don't end up going off on a tangent.

Being a voluntary effort, people are probably going to first port stuff they need. For my current situation, trying to make use of the modbus_tk third party python library, I have discovered that I need the struct module, as well as the ability to import stuff (as highlighted in #258) so that I don't have to have all my python code in one file.

Another approach to supporting stuff like communication protocols would be to port existing C libraries and make use of them in python. This is probably a quicker solution in the short term?

Either way I would really appreciate some guidance. I'm sure any effort invested in this area will be well worth it in the end.

@pfalcon
Copy link
Contributor

pfalcon commented Feb 14, 2014

This is big topic, so just few IMHO notes:

  1. One issue with having "standard library" is that once you add stuff there, you can't take it away, or people will count that it's there and blame you when it's not. Currently there's no "standard library" going along with uPy, and I hope it will stay that way for as long as possible. (Obviously, that doesn't mean that people should be denied to use standard modules - just that they should bring stuff they need themselves, and there should be tools to help that).
  2. In first part, you argue for "top to bottom" approach - a committee sits and labels thru CPython modules, then appoints project managers and developers to execute it. But as you say yourself, there's no committee, and surely no project managers and developers to back such effort. Also, labeling 100 modules doesn't bring even single of them closer to uPy.
  3. On the other hand, "bottom to top" approach as described in 2nd half is what looks sound, I must say, after practicing it for more than a month. Just set very specific aim, and go towards it, resolving small, realistic issues. I posted about this approach here: https://groups.google.com/forum/#!topic/micro-python/6YPL8dvgywk , inviting people share their experiences.

You're very right that struct should be one of the most useful and among first one to implement. But it's closely related to array module (they deal with same binary types) and FFI (which have to deal with variable & pointers of the same bin types, and ideally, with structures too). So, it requires some thought and refactoring to cover all them with minimal code. Today I landed some code which I believe is generic enough to cover all them: 8bc3516 . So, work on struct module is welcome.

Another approach to supporting stuff like communication protocols would be to port existing C libraries and make use of them in python. This is probably a quicker solution in the short term?

Then we'll have python capable only of executing do_stuff() function, which implements everything else in C. So no, the best way to get Python working is to exercise it, report issues, submit patches, etc.

@royemmerich
Copy link
Author

This is big topic...
...
...just that they should bring stuff they need themselves, and there should be tools to help that

Considering how huge this task is immediately got me thinking how best to divide it up. I agree that people should be willing to muck in and help by implementing the stuff they need. In order for these voluntary efforts not to be wasted, we need to set up a proper framework and the accompanying documentation for people to follow. If we don't do this, adding standard libraries will just end up a big mess which will only serve to drive new people away.

In first part, you argue for "top to bottom" approach

I'm not in favour of any particular approach. My initial post was just a first pass at figuring out a way to manage this task. What I am trying to do is make things clear so that everybody can easily see what functionality exists and what still needs to be added. Maybe we need to have somewhere where we can host an automatically generated copy of the API documentation based on the current version in github (e.g. with sphinx and https://readthedocs.org)

Then we'll have python capable only of executing do_stuff() function, which implements everything else in C. So no, the best way to get Python working is to exercise it, report issues, submit patches, etc.

Yes I would also prefer writing as much of my code in python as possible. In fact I think this was one of @dpgeorge's initial reasons for starting this project. Why I suggested implementing the MODBUS RTU protocol in C and using it in micropython is because it would initially be quicker to implement. One would just have to take an existing MODBUS library written in C, add a few python binding methods and you'd be ready to go (that is what my inexperienced gut feel tells me is required).

Taking the python route by making use of the modbus_tk library, one still has to implement the struct and array standard libraries. As I really need to make progress with my datalogger and considering the fact that I don't feel confident proposing a reasonable C style framework for supporting the development of python standard libraries, I thought I'd start this thread to try and get things moving along.

@pfalcon
Copy link
Contributor

pfalcon commented Apr 26, 2014

Concrete steps along this discussion were made in #405, and can be further discussed in http://forum.micropython.org/viewtopic.php?f=5&t=70 . Closing.

@pfalcon pfalcon closed this as completed Apr 26, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rfc Request for Comment
Projects
None yet
Development

No branches or pull requests

2 participants