This implements #3412 . Besides what's written in the commit message, this is supposed to be a universal answer to "can we enable something in the Unix port?"

I know there's already an abdundance of config options, but maybe it would be nice to be able to do make dev ALSO_ENABLE_ALL_VFS_STUFF=1 instead of having to modify mpconfigport_dev.h or keeping an extra branch for that?

So, this is definitely RFC. VFS stuff commented out because I wasn't sure about state of it (yeah, there're too many things for I forget them too, we need more docs %) ). The idea is to have "all" (perhaps, almost all) enabled, including VFS.

VFS stuff commented out because I wasn't sure about state of it

Like, I quickly made this commit just as a proof of concept, and commented it out to be 100% sure it won't interfere with normal POSIX fs handling. Hoped to get back to it, but didn't (partly because there's no response to other PRs), so just posting as is.

Sorry for the delay here. I do think this is a good idea, to have a separate PC/desktop executable for development. I would like to see it mirror bare-metal ports as much as possible, eg use extmod/moduselect.c instead of ports/unix/moduselect.c, and maybe even extmod/modlwip.c via a tap/tun device (although that might be difficult and impractical).

But how about going the other way: make the "micropython" executable the full-featured one, and introduce a new, minimal-but-functional PC/*nix executable? This would be a good (and perhaps last) chance/opportunity to do this, to pick a shorter name for it (revisiting #17), and immediately disable some non-necessary things like btree (since ffi can be used to load an sqlite shared lib, for example).

Thanks for reply!

I would like to see it mirror bare-metal ports as much as possible

Well, to clarify, mirroring bare-metal ports is not the intended primary purpose of the proposed variants, but rather be more convenient for development/experimentation. And in general, both Unix and baremetal ports to a "sweet spot", to be worked on, to improve their interoperability. (And again, baremetal ports need to be changes to achieve that, not just unix).

eg use extmod/moduselect.c instead of ports/unix/moduselect.c, and maybe even extmod/modlwip.c via a tap/tun device (although that might be difficult and impractical).

Again, "instead" is not the intention of the proposed micropython-dev. In addition - certainly. Changes would need to be done to some parts to make that viable. E.g. extmod/moduselect.c is not adequate with its bust-polling/O(n) approach. Instead, it should be reworked to queue up IO-ready objects to make them immediately accessible. And of course, just interfacing that with native POSIX fd's will be a task on its own.

But how about going the other way: make the "micropython" executable the full-featured one, and introduce a new, minimal-but-functional PC/*nix executable?

Besides causing unneeded disruption, that also undermines all work done so far on Unix port. The current "micropython" executable of Unix port is such minimal-but-functional executable, and was always intended to be. It's also the reference MicroPython port per functionality provided, and the best port, given the uselect module deficiency in all other ports mentioned above, questionable features like module weaklinks, and general attitude of other ports like "oh, we have a lot of flash yet, let's add yet another useless feature".

to pick a shorter name for it (revisiting #17),

Who's calling for that? Everyone got used to it long ago, and it turned out not too bad, on my pretty busy Ubuntu system there're no other "micro..." executable, actual autocompletion from "mic" already works, and it definitely makes the executable recognizable and self-documenting.

and immediately disable some non-necessary things like btree (since ffi can be used to load an sqlite shared lib, for example).

btree is the absolutely necessary thing, it's what makes MicroPython a full-stack language! It can't be a complete language without offering data storage/querying capabilities, and btree is the only thing which works on any port - and again, portability and consistency among all ports was always an important goal for MicroPython. It's a great shame that the btree module is underused, shows how polarized MicroPython community still on led-blinking. In "big world", any webapp almost certainly goes with a database, and there're a lot of webapp developers around, MicroPython just didn't reach them yet (those who are interested in unbloated solutions).

This would be a good (and perhaps last) chance/opportunity to do this,

So, I don't see any convincing reasons to do any shattering-like changes to the existing "micropython" executable. Actually, the whole idea of "micropython-dev" is to avoid them, yet not hold back people who're interested in more advanced things like profiling, AST accessing, etc.

Main Unix port of course needs further elaboration, to improve compat with baremetal ports, e.g. #4308 , and I'm ready to implement for uos.rename(). (And that's about it, as usual, any additions are strictly rationed.)

It's also the reference MicroPython port per functionality provided

Being the reference port for functionality actually sounds like an argument for having all functionality by default as dpgeorge suggests, instead of against? I don't know who stated the original intent of the unix port, or where. I'm also not sure if that matters. For example suppose we do a survey and turns out the most-used scenario across all users is firing up the unix port to test some new behaviour by enabling options in mpconfigport or on the commandline. Should that be the case then whatever the intent ever was would be outdated and having everything on by default should be the new intent. Then again, there's no survey hence no knowledge of actual use so an informed decision is rather hard to make. But I kind of like the idea of having it all by default just because it matches my particular usecase and can imagine I'm not alone with that.

Being the reference port for functionality actually sounds like an argument for having all functionality by default as dpgeorge suggests, instead of against?

Of course not! Reference port has as little as possible functionality as required to write full-fledged, portable MicroPython apps, not as much! This should be quite obvious, because you can bloat up something ad infinitum, but reduction always has its limits. If that's not obvious, then smart people taught us about it:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

https://www.brainyquote.com/quotes/antoine_de_saintexupery_103610

I'm also not sure if that matters. For example suppose we do a survey and turns out

Suppose we did a survey and find out that in some group the most common use for newspapers is wrapping greasy things in it. What does that tell us about history of alphabet, Gutenberg, journalists imprisoned or killed trying to bring information to the public, etc.? Does all of that matter is we just made a conjecture that the purpose of newspaper is being a wrapping convenience?

Looks like a difference in understanding of the meaning of 'reference' :]

Wrt your analyogy: nice idea, but could have been more to the point. Both cases possible (unix port has everything vs has it partly) are still both subsets/specialisations of the general usecase. Reading a paper vs using it as a wrapper are not both subsets of it's primary use. Reading a paper just for the comics would have been closer :P

Looks like a difference in understanding of the meaning of 'reference' :]

"Reference" is one you test against for compliance. Testing an app against Unix port quick catches cases like usage of "os", "socket", etc. (don't exist in MicroPython core API, only as extensions), "uos.listdir" (ditto, in the core API only "ilistdir"), etc, etc. And my understanding is also consistent with https://github.com/micropython/micropython/wiki/ContributorGuidelines and existing status quo. So, we seek how to extend the current picture of the world, not to shatter it.

Wrt your analyogy: nice idea, but could have been more to the point.

Indeed, was absolutely random one, and could be worse ;-).

FWIW, this was updated to enable VFS, and dev_test make target added.

Updated to link micropython-dev with -rdynamic, to allow to resolve API symbols in the executable (either for ffi or dynamic modules).

with -rdynamic,

Interesting to see this works. I never figured out how to use this and not get linker errors for fatfs, apart from just disabling it completely.

Heh, I remember that issue too. Well, somehow currently it builds, I don't know what's wrong ;-).

Looking at what I tried I think it either has to be disabled completely (default in mpconfigport.h, but the unix Makefile cannot add oofatfs/ff.c etc to LIB_SRC_C which is now the case unconditionally, else there are unresolved references which pop up only with -rdynamic) or enabled completely (in which case mpconfigport.h also needs MICROPY_VFS and MICROPY_VFS_POSIX next to MICROPY_VFS_FAT, and also needs to define mp_type_textio). Anything else probably won't work.

Yep, probably something like that. Of course, that needs to be fixed. Digging in my git stash, here's what I used to get around it:

--- a/lib/oofatfs/ff.h
+++ b/lib/oofatfs/ff.h
@@ -258,7 +258,7 @@ typedef enum {
     FR_INVALID_PARAMETER    /* (19) Given parameter is invalid */
 } FRESULT;
 
-
+#define FRESULT __attribute__ ((visibility ("hidden"))) FRESULT
 
 /*--------------------------------------------------------------*/
 /* FatFs module application interface                           */
@@ -293,6 +293,8 @@ FRESULT f_umount (FATFS* fs);                                       /* Unmount a
 FRESULT f_mkfs (FATFS *fs, BYTE opt, DWORD au, void* work, UINT len); /* Create a FAT volume */
 FRESULT f_fdisk (void *pdrv, const DWORD* szt, void* work);         /* Divide a physical drive into some partitions */
 
+#undef FRESULT
+
 #define f_eof(fp) ((int)((fp)->fptr == (fp)->obj.objsize))

A micropython-dev build was added in 2357338