Replies: 9 comments 11 replies
-
|
mounting and reaching thos files would assume that you know AOT all files paths because the server won't likely give us a list of files to crawl so it would be a poor mount FS experience, imho, but the thing is, PyScript gives you all FS primitives to created directories and fetch files as binary into those. True that we don't expose an explicit helper/utility but that's all we do internally on bootstrap via the config ... as summary: would a basic API such as About persisting files via GitHub, I have successfully used gist-fs to do that but the name of the gist and/or the token must be kept as private as possible, which is the common bummer of the Web, something I've started working on but not solved yet to date. If you carefully avoid potential 3rd party malicious script or users out there to see and retrieve your authentication token though, you can go pretty far with that and use that as js_module in your PyScript project. |
Beta Was this translation helpful? Give feedback.
-
|
forgot to mention we also already have the ability to unzip or un-tar.gz archives out of the box via config and files, so I wonder why that wouldn't work for you 'cause we use it already without issues (but tar.gz is preferred over zip simply because it doesn't need 3rd party libraries). |
Beta Was this translation helpful? Give feedback.
-
|
@WebReflection. The short answer yes,I am getting really confused , I think you have misinterpreted what I was saying. The Config files doesn't work during run time. I might have used mount wrong. I was thinking, It does what files does in your config . I renamed it mount because it sounded better and that's how zfs handles it ; but save would be a better name and adding a third perimeter would allow it specify how to store the files gotten by the server and how to handle it in your pyscipt-vfs(stored in webstorage ,vfs , or other place (a SQL server maybe). I am only (at the moment)only think one way fetching data from a server I wasn't even think about pushing it back to the server
Since all the files are in a zip we only deal with 1 file we are just decide how and where where extracting it at run time. What you suggested for an API probably would work if handles , zip ,bzip, someways it would be nice if it could handle iso but that probably not worth your time.. I thought you could add a third perimeter on how to store it , index.db chrome fsapi etc. Something you should know right now. That you might have missed that Is very important is I am zipping my whole project up and extracting it through your file config option in the lib directory This in theory should get around having to put each python file i am using in my pyscript-config.toml( or whatever is called). I couldn't find anything mentioning on dealing with the file system in your user guide. I know pyodide does have some documention but I am using micropyhton. You may have it somewhere documented but this type of thing should really be in you user guide, and as you said you already have the capability I just want to expanded a little bit and have it documented. |
Beta Was this translation helpful? Give feedback.
-
so here it's me being confused ... as I don't fully understand what you are asking but please note that the virtual filesystem is provided by Emscripten, not pyodide nor micropython, it's always the same VFS and you could have it also available in JS via https://github.com/WebReflection/fs or other ports, as example. About the API, the exposed VFS and Python normal ways to deal with a file system are the same, it's the unzipping that is maybe missing out of the box (requires a library) but tar.gz is available and usable natively in both Pyodide and MicroPython, yet we don't provide a simplification of it over the network. If I might: are we saying you'd like to just grab a zip and extract it at runtime over your virtual MicroPython or Pyodide virtual FileSystem? If that's the case, I don't see why we shouldn't provide such utility that works also with regular files ... as in move or fetch_file or something to eventually discuss and shape while all internals are already capable of doing that. |
Beta Was this translation helpful? Give feedback.
-
|
so here it's me being confused ... as I don't fully understand what you
are asking but please note that the virtual filesystem is provided by
Emscripten, not pyodide nor micropython, it's always the same VFS and you
could have it also available in JS via https://github.com/WebReflection/fs
or other ports, as example.
I was not aware of https://github.com/WebReflection/fs (I'm going to have
to look at that) but I am aware that pyodide and micropyhton uses
emscripen( Wich is a build tool with an API) but I don't know how to
access emscripen API. I only said py-script VFS to be clear I am talking
about the virtual file system py-script uses I could have said emscripen
VFS
If I might: are we saying you'd like to just grab a zip and extract it at
runtime over your virtual MicroPython or Pyodide virtual FileSystem? If
that's the case, I don't see why we shouldn't provide such utility that
works also with regular files ... as in move or fetch_file or something to
eventually discuss and shape while all internals are already capable of
doing that.
yes ,you may (it not a problem I am more worried about you getting annoyed
by me.)
Basically yes that exactly what I want but with more control and more
options (if you don't mind). It would be nice to be able to use the API to
make the home directory be stored in web storage or using the chrome
filesystem API you could even make it be able to use Dropbox or another
online storage. Those are just ideas maybe not doable but mainly I just
want to fetch a file from a url and extra it to the emscripen (what
ever)VFS which is what file is doing in py-script config. I should have
given you an example so I'll do that now (I update my original commit it there now.)
I updated this for better readability(I hope).
|
Beta Was this translation helpful? Give feedback.
-
heh ... let's take one step at the time ... the OPFS is something that needs investigation but we want to provide that ability sooner than later, remote services such as Dropbox are an entirely different topic as just connecting to those is problematic if you need to store credentials in a config file ... we don't want people to store private fields or credentials in their config, we offer a SECRETS like reverse-proxy in PyScript .com but even there, we need to be careful about what's possible and what isn't ... so, back to this request that The persistent part of affair would probably come later but persistent files at runtime might be more problematic to address (the config is a one-off thing, runtime behavior is a different beast, we already saw that with the recent micropip locked/frozen cache too). Let's see how this moves forward, but I think as long as we are aligned to what's possible, what's not secure, and what's desirable, I think this is a good feature request 👋 |
Beta Was this translation helpful? Give feedback.
-
|
Thank you I would be looking forward to whatever you decide. I only mentioned things like Dropbox ,git client just to keep that in mind while design the API. Actually with above in place I, think we probably could make those a plugin but as you said one step at a time. The export function does indeed do something at least as yesterday with out it I would not be able to zip my project on py-script.com. I found it in the config of a new created project the comment above if says something to the like of .... and that it , maybe I shouldn't have said undocumented, it sort of documented,I can't expect much from beta. very interesting anyways, I don't think it meant to work like I am using it but it is very handy. |
Beta Was this translation helpful? Give feedback.
-
|
can you please tell me where that |
Beta Was this translation helpful? Give feedback.
-
|
also, we cannot keep Dropbox in mind with current API because we have no way, or mean to, expose users' credentials out there ... that API is going to be different and not necessarily a 1:1 PyScript API thing as it would require different steps before ever working |
Beta Was this translation helpful? Give feedback.
-
|
I Am not concerned about Dropbox. The export option is not in the files section so it is probably not in the py-script stack (I had my config written wrong I fixed it above now ). the export probably comeing from the ide from py-script.com. Py-script allows to get extra arguments from there config (at first I thought it might have been from the stack) . I found the export option on the first line when starting a new project. I tried porting my code to gitlab and it doesn't do anything there but on py-script.com it puts an export.zip file in my project. Could you give a link to the code freeze bug you had talked about I would mind finding out more about what you are talking about.
I want to spend just amount of time understanding you (if not more) than you spent on me. I would think that type of thing would not have been possible with the way you use promises.
I update this for better readability. |
Beta Was this translation helpful? Give feedback.
-
|
btw, from the Pyodide API it looks like they have a persistent FileSystem mechanism that works only in Chrome so far, but might give you both an FS API and persistency. It's beyond the Python API and it's exposed via the JS API nly but that might be our starting point to have persistent FS in PyScript too. |
Beta Was this translation helpful? Give feedback.
-
|
@WebReflection .I did not say a proper Thankyou last time for spend so much time I greatly appreciate your patience and support I know I am hard to deal with and value all the links you gave me. |
Beta Was this translation helpful? Give feedback.
-
|
I want to turn this into an issue but I would like to know what i would need to do this correctly yet . That was the main reason for the redo on the first commit. If anyone does work on this can they give me a link or let me know I would like to try all betas and keep an eye on the progress. |
Beta Was this translation helpful? Give feedback.
-
|
Tested a persistent fs today using opfs and it worked well with micropython on osx(chrome, firefox, safari), android(chrome) and ios(safari). Just aded a proposal for it and saw that post. It will be great to have this as official module in pyscript. |
Beta Was this translation helpful? Give feedback.
-
|
i am an writer not an developer and needed to dig a little deeper into pyscript to find |
Beta Was this translation helpful? Give feedback.
-
|
@Ridensium nice job! We'd love feedback on how we could make this easier (docs?). |
Beta Was this translation helpful? Give feedback.
-
|
@ntoll thanks, will give feedback sure. Just want to make some tests. While plaing with opfs saw that its file read/open is 100 times faster than the native one in micropython (for 5mb text file 0.035s vs 2.27s on osx/safari for example) which is really worth it to be considered for use apart from the persistence. As well wanted to push both interpreters up to the limits. Check this out: pyodide interpreter made with 10mb chunks 5010.0 Mb file in opfs for 2.7s ;) yep 5Gb indeed |
Beta Was this translation helpful? Give feedback.
-
|
I would like to say, I always love doc and and very untechnical so anything you write i would like to read. I am also very excited to see the work you guys are putting into this. There a lot that is confusing and i have some suggestions. To give you some example there's really about 3 type of files in the file systems files on the server, file and cookie storage on the client sandbox that not visible outside of the browsers (if i am not mistaken) and the third client sandbox directory that's visible out of the browser.. The current api documention does really talk about it that well at least for me but then again I have that type of trouble. The thing with the persistent file system is also confusing how It can work on many browsers but the file system API which is part of does not. |
Beta Was this translation helpful? Give feedback.
-
|
I see it was discussed already in here, well, I have same Nicholas questions around how can we improve or rather what is it that is missing in our case ... but maybe I need to just look closer at your demo? |
Beta Was this translation helpful? Give feedback.
-
One of the things I have problems with is too much info. This is why I am constantly updating my first post, it is an attempt to include a summary of all thing in this discussion. (so you only have to read the first post).
The problems with the way pyscript handles VFS now.
Problem 1 It is hard to load files into the VFS.
Suppose I want to create a library on pyscipt.com. This is what I was
doing.
I created a directory myLib and in that directory I have 2 python scripts.
let's call it myFirstGoup.py and mySecGroup.py it looks something like this
ls myLib
now suppose I have a main.py file in the top level directory with contents
You have to edit my pyscript.toml config and put these
two files in the config so it would look something like this
as you can see this is problematic (as a file grows more and more files are
need to add to the config)
Problem 2 The files are not persistent.
If I wanted to make a open source pyscript editor like the one found on pyscript.com, I would some how have to store my files; currently, there no easy way to do this. Even though emscripten have a mount option for persistent-data there no documented way to access this api.
Problem 3 You can not modify files during run time though the JavaScript API.
Again, use the example from problem 2 ; If I am going to create an open source editor that support git the library from isomorphic-git require the node like fs api.
Problem Summary
Solution and work arounds
Problem 1 (It is hard to load files into the VFS) partial solutions
(At the time of this writing 2-22-25 there is an export option that allows py-script.com to zip up you project it says this)
# Bundle your project into a zip file which can be downloaded or used#as a module in your Anaconda Toolbox Excel Add-inyou can take export.zip and put it in the VFS so you only have to extract it once. Use the example above put something like this in your pyscript.toml
problem 2 (The files are not persistent )partial solution 2
you can use
problems with this approach
Current status
Key take away
I been doing a lot of thing that require modifying the files on the vfs some more examples
Beta Was this translation helpful? Give feedback.
All reactions