tex cache lockfile retries should be configurable #7776
Comments
|
Rather, the locking should be done per-entry, rather than globally for the whole folder, as I don't think writing to separate cache entries creates any problems? The current lock implementation can only lock folders though (because it writes to a file within that folder). We can either move each cache entry into its own subfolder (not too hard?), or change the lock implementation to be able to lock single files (off the top of my head, one should be able to do so by creating lockfiles named |
|
Another option would be to isolate your process's tex cache from each other by launching your python processes as MPLCONFIGDIR=/tmp/some_pathN python foo.pywhich if your |
|
Isolating the processes tex cache does not work when e.g. using multiprocessing (at least not to my knowledge). Therefore the suggestion to isolate each processes tex cache does not help. So either the tex cache dir should be configurable from within a running process, or the lock implementation should be changed. Th problem actually gets worse the more cores/processes are used. Seen this at 32 vs. 16 cores. |
|
This is a dirty hack to fix the issue meanwhile: Note that if, for some reason, matplotlib is already loaded before a subprocess (in my example from joblibs) wants to use matplotlib, then just setting the environmental variables alone is not enough anymore, since the TexManager has already established it's cache directory. Using different temporary directories for matplotlib will also speed up plotting significantly. For me, it was about 10x the speed (AMD Threadripper 1950X, 16C, 32T, plotting 3432 plots to an SSD). Note of course, that the subprocess_task is best used as long as possible. Each process needs to set-up matplotlib this way only once, not for every call to the task. |
|
Should be mostly closed by #10596. Please request a reopen if that turns out not to be enough. |
Matplotlib version 3.2.1Summary:I am trying to plot several plots in parallel (using a scheduler to call python several times) using with When I try to create the plots sequentially, I am able to do so without any problems. Is there any way I can disable this lock file or generate a new one every time I run my plotting script? Error:Short:Full log: |
|
I suddenly get this error now after updating to matplotlib 3.4.2 from version 3.2.1. Note that it happens when importing matplotlib with This is on a CI/CD build, so there might be other processes around using matplotlib. Not sure if there are any changes in matplotlib related to this error, but I never got it before. |
I am doing data processing and generating many plots with many scripts running in parallel. The scripts use texmanager, and many fail when drawing to
pngdue to below. My plots are large, with many subplots, and the filesystem I'm writing to is slow. Therefore, the hardcoded limit of 50 tries with 0.1 second waits is not appropriate.Those hardcoded values should be configurable so that the user can change them if needed.
Unfortunately, the nature of the problem makes it hard to write a minimum code snippet that reproduces it, because it only occurs with big plots when running many processes in parallel.
I'm using Matplotlib 2.0.0rc2 (installed through pip) on Python 3.5.
The text was updated successfully, but these errors were encountered: