Skip to content

Delay the fc-list warning by 5 seconds #7532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from

Conversation

astrofrog
Copy link
Contributor

@astrofrog astrofrog commented Nov 29, 2016

In #5836 (comment), @tacaswell said:

If there is a non-thread based way to get a 5s delay I am 👍 on that, but bringing threads in this seems like complexity overkill.

The solution here doesn't use any extra threads/processes compared to what was being used before, and is a pretty simple/robust solution.

cc @tacaswell @efiring

@astrofrog astrofrog force-pushed the fc-list-warning-delay branch from 4f1ca3f to 2a90d95 Compare November 29, 2016 20:40
@efiring
Copy link
Member

efiring commented Nov 29, 2016

timeout.communicate() has a timeout kwarg only in python 3; at least, I don't see it in the docs for python 2.7.

@dopplershift
Copy link
Contributor

In that event, pass a kwargs dict to communicate and make it empty on Python 2. It would be very nice not to see this warning constantly--since I work on Python 3.

@astrofrog
Copy link
Contributor Author

Aw man, today is my day of implementing things and then finding out later they don't work in Python 2... But yes, I think we could implement this at least for Python 3 users?

@QuLogic
Copy link
Member

QuLogic commented Nov 29, 2016

We optionally use the subprocess32 backport, which I assume includes the timeout, so don't change the original code, but change the compat code to remove that argument if necessary.

@astrofrog
Copy link
Contributor Author

So in principle it's also possible to implement a solution based on writing a loop and using p.poll, something like:

        import time
        start_time = time.time()
        while time.time() - start_time < 5:
            if p.poll():
                output = pipe.communicate()[0]
                break
        else:
            warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
            output = pipe.communicate()[0]

But this doesn't seem as tidy. What do people think?

@dopplershift
Copy link
Contributor

Ick. timeout seems tidy and the way to go IMO.

@rmjarvis
Copy link
Contributor

There is a much simpler implementation I think using signal:

import signal

def get_fontconfig_fonts(fontext='ttf'):
    """
    Grab a list of all the fonts that are being tracked by fontconfig
    by making a system call to ``fc-list``.  This is an easy way to
    grab all of the fonts the user wants to be made available to
    applications, without needing knowing where all of them reside.
    """
    fontext = get_fontext_synonyms(fontext)

    def delayed_warning(signum, frame):
        warnings.warn('Matplotlib is building the font cache using fc-list. '
                      'This may take a moment.')

    fontfiles = {}
    try:
        signal.signal(signal.SIGALRM, delayed_warning)
        signal.alarm(5)
        pipe = subprocess.Popen(['fc-list', '--format=%{file}\\n'],
                                stdout=subprocess.PIPE,
                                stderr=subprocess.PIPE)
        output = pipe.communicate()[0]
        signal.alarm(0)
    except (OSError, IOError):
        # Calling fc-list did not work, so we'll just return nothing
        return fontfiles

@astrofrog
Copy link
Contributor Author

astrofrog commented Nov 30, 2016

@rmjarvis - I agree that looks nice, so it just depends whether people are happy to have an additional thread (which I think your solution requires behind the scenes).

@tacaswell - do you have any opinions on this?

@rmjarvis
Copy link
Contributor

Also, I think signal doesn't work in Windows, so the full solution would probably require wrapping that in a try/except block and either always or never emitting the warning in Windows systems.

@dopplershift
Copy link
Contributor

Then I'd argue strongly that the simple timeout kwarg is the right way to go; we at least have a timeframe for when Python 3 will be the only one supported.

@efiring
Copy link
Member

efiring commented Nov 30, 2016

I agree with @dopplershift; using signals can get tricky and dangerous (and the sample code provided is incorrect because it does not reset the signal handler on exit), and applying that big hammer to this flea of a problem is not appropriate or acceptable for mpl. So it looks like timeout kwarg or nothing--and nothing is still a good option.

Side note: python3 subprocess is using threads.
Side note 2: fc-list is only run on unix-like systems, not on Windows.
Side note 3: To me, the more important thing to fix is the unnecessary second call to fc-list. If the problem is that sometimes this call takes long enough to make the user wonder what is going on, then the first thing to do would seem to be to make sure it is only called once, not twice.

@dopplershift
Copy link
Contributor

dopplershift commented Nov 30, 2016

Solving the double message would be nice, but I think it's only slow once, since after that it has cached results?

IMO, though, it would be nice regardless to only see the warning if we're actually going to end up waiting, so regardless I'm 👍 on adding a 5 second pause before actually issuing a warning.

@efiring
Copy link
Member

efiring commented Nov 30, 2016 via email

@efiring
Copy link
Member

efiring commented Nov 30, 2016

Elaborating on @QuLogic's comment: yes, we use subprocess32 with python 2.7, but I think the problem is that although its communicate accepts a timeout argument, it is not supported in the way we need until python 3.3, when the argument was officially added:
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate

@tacaswell
Copy link
Member

👍 to this solution 👍 to only fixing it for python3 if it is non-trivial to get it working in py2.7 👎 to signals 👎 to poll

👍 to re-writing the font finding to only need one call out, but that is a possibly bigger project.

My main protest to threads is mpl having to maintain the thread related code. If upstream is providing an interface that hides the threads, I trust them 😈 .

@tacaswell tacaswell added this to the 2.0.1 (next bug fix release) milestone Nov 30, 2016
@astrofrog
Copy link
Contributor Author

I've made it so it now only does this on Python 3. The solution to pass a kwarg dict to communicate doesn't work because for Python 2 we need to then emit the warning before the first communicate call.

efiring
efiring previously approved these changes Nov 30, 2016
Copy link
Member

@efiring efiring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it should work; not so easy to test.

try:
output = pipe.communicate(timeout=5)[0]
except subprocess.TimeoutExpired:
warnings.warn('Matplotlib is building the font cache using '
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: I would put this message in a local variable so that it doesn't need to be spelled out twice in the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good - done!

@astrofrog
Copy link
Contributor Author

The easiest way to test it locally is to replace the fc-list command with e.g.

        pipe = subprocess.Popen(['sleep', '10\\n'],

@NelleV NelleV changed the title Delay the fc-list warning by 5 seconds [MRG+1] Delay the fc-list warning by 5 seconds Dec 1, 2016
output = pipe.communicate()[0]
# We emit a warning if the process has not completed after 5 seconds.
# We avoid showing it before since if this takes < 5 seconds, the user
# probably won't notice. However, this only works on Python 3 because
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

subprocess32 doesn't have this? It is only an optional requirement, but that means that if people read our installation file (…), it'll work for 2.7

message = ('Matplotlib is building the font cache using fc-list. '
'This may take a moment.')

if six.PY2:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't test for python2 but for whether subprocess has the attribute TimeoutExpired here:

hasattr(subprocess, TimeoutExpired)

subprocess32 being an optional dependency, that may work for python2.7.

@NelleV
Copy link
Member

NelleV commented Dec 1, 2016

Apart from my comment, this looks good.
Indeed, not so easy to test :)

@QuLogic
Copy link
Member

QuLogic commented Dec 1, 2016

@efiring subprocess32 is based on the 3.2 module, but like many backports, it received continuous updates. The latest release (and presumably anything back to 3.3-equivalency) does include timeout, so as @NelleV says, we shouldn't skip this solely on the basis of running on Python 2.7.

@@ -26,7 +26,7 @@
else:
import subprocess

__all__ = ['Popen', 'PIPE', 'STDOUT', 'check_output', 'CalledProcessError']
__all__ = ['Popen', 'PIPE', 'STDOUT', 'check_output', 'CalledProcessError', 'TimeoutExpired']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be line-wrapped ❤️ pep8

Copy link
Member

@tacaswell tacaswell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yet more python2 compatibility issues.

Either this needs to be shimmed out more cautiously or subprocess32 needs to be a runtime dependancy.

@@ -36,6 +36,7 @@
STDOUT = subprocess.STDOUT
CalledProcessError = subprocess.CalledProcessError
check_output = subprocess.check_output
TimeoutExpired = subprocess.TimeoutExpired
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work on python2 unless subprocess32 is installed.

Python 2.7.12 (default, Nov  7 2016, 11:55:55) 
[GCC 6.2.1 20160830] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.TimeoutExpired
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'TimeoutExpired'
>>> 

This looks like it is working on travis because subprocess32 in an install require (not a run-time require) so it is always getting installed on travis. I think that there is a way to install mpl such that this will fail.

@jenshnielsen
Copy link
Member

We cant really make subprocess32 a full dependency, it's explicitly listed as not tested on windows see https://pypi.python.org/pypi/subprocess32/

@astrofrog
Copy link
Contributor Author

Sorry for the delay, today is a feature freeze day for astropy so things are pretty busy there. I can get back to this next week, or if any of the maintainers want to push commits to this branch, feel free to.

@anntzer
Copy link
Contributor

anntzer commented Dec 5, 2016

There's https://docs.python.org/2.7/library/threading.html#timer-objects (available both on Py2 and on Py3). Something like

timer = Timer(5, lambda: warnings.warn("the warning message"))
timer.start()
try:
    <run fc-list>
finally:
    timer.cancel()

should be good enough?

(and you can cancel already expired timers)

@astrofrog
Copy link
Contributor Author

The Timer approach does seem cleaner - any objections to changing to that?

@QuLogic
Copy link
Member

QuLogic commented Dec 7, 2016

That seems like no objections, but I'd squash out the other commits if we aren't going to use them.

@QuLogic QuLogic changed the title [MRG+1] Delay the fc-list warning by 5 seconds Delay the fc-list warning by 5 seconds Dec 7, 2016
@QuLogic QuLogic dismissed stale reviews from efiring and tacaswell December 7, 2016 02:53

Changing implementation.

@anntzer
Copy link
Contributor

anntzer commented Dec 9, 2016

See #7596 for an alternative implementation.

@anntzer
Copy link
Contributor

anntzer commented Dec 12, 2016

The original issue was solved by #7596.

@anntzer anntzer closed this Dec 12, 2016
@QuLogic QuLogic modified the milestones: 2.0 (style change major release), 2.0.1 (next bug fix release) Dec 12, 2016
@astrofrog
Copy link
Contributor Author

@anntzer - thanks for wrapping up the alternative implementation!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants