Some clarification:

RcParams

• There are parameters verbose.level and verbose.fileio which are inside mpl.rcParams. It seems those are not actually used anymore or at least deprecated. However, they are not part of mpl._all_deprecated. I left them out of the template and the test, but should the actually be added to the mpl._all_deprecated?

• There is a docstring.hardcopy : False in the file. When uncommenting, this will raise a warning. I have no idea what this parameter would even do. For the moment I left it in the template file, but commented with ## such that tests run smoothly.

• The following parameters are inside of mpl.rcParams, so I added them to the template despite me not having any good explanation for them:

  pdf.use14corefonts : False
  pdf.inheritcolor : False
  pgf.rcfonts : True
  pgf.preamble : []
  pgf.texsystem : xelatex
  pgf.debug : False
  text.latex.preview : False
  animation.embed_limit : 20.0
  path.effects : []
  

  This might still be done once someone feels like they know a good comment string to add.

Tests

I added two tests:

• Test if all rcParams are in the template file This should raise in case there is an rcParam in the mpl.defaultParams, which is not found in the template file. All params in mpl._all_deprecated and
  mpl._deprecated_set are excluded from the test. Also all params starting with _ and those with "verbose" (see first item). This test will hopefully force people adding new rcParams not to forget to add them to the template file as well.
• Test if the uncommented template is a valid rc file. To test this, the format of the template file has been changed slightly. All rcParams are are commented with a single #, while true comments are prepended ##. This allows to read in that file, remove single leading # and read it in as rcparam file. This is done via mpl.rc_params_from_file. This would raise a Warning for every invalid line in the file. Since fail_on_error=True does not raise an error but only a warning, this test will probably never fail, but raise a bunch of warnings. Please comment on whether this is the desired way of performing such test.

What does codecov/project/tests — 98.35% (-0.04%) compared to 56a6b9b failing mean?

pytest has a context manager to capture warnings, I suggest using that and then asserting that there are no warnings.

The coverage report says that the coverage on the test suite has gone down, are you sure the new tests are actually running (pytest is always a bit mysterious to me)?

1. Ok, I changed it to raise an error if any warning happens.

2. No, I'm not sure about anything here. ;-)
   What is "coverage"? How would one find out the test is run? I was under the impression that any function starting with test_ would be run?

I tend to fall back to putting assert False in locally and making sure the tests fail when I expect them to.

My test only covers 0.01% of the all the code, simply because it is only written to test one specific thing. I think I'm misunderstanding something fundamental here.

Where exactly do you want me to put assert False and for what reason? I have made sure the test does what I expect it to do locally on my computer, is that what you mean?

So if there is anything you want me to do for this to successfully "cover" something or to be run at all, just tell me what it is. Feel free to take your time and do so whenever you're less busy since rushed one-sentence answers do not usually help much in such cases.

I am still not exactly sure how the test coverage has gone down. Is the size of the codebase including the test suite itself?

The tests with assert False both failed as expected with an AssertionError on travis. So that means that they are indeed run.

Should probably also clean out the debugging prints and other commented out things. But I am, in principle, +1 on this once we confirm that everything is working.

The commented statements that are still left in the code are there by pupose: I was thinking that if someone has this test failing for his commit they would immediately direct him to the source of the problem.
Of course I might delete if you want.

That needs a big fat comment in the matplotlibrc.template, then, right before the prop_cycle line, pointing out that it is exceptional, and is the only place in the file where quotes are allowed, and that they are indeed required there.

Thanks @efiring for coming back to the issue of string escapes. I brought his up here, but did not get any feedback, so I assumed this to be ok, as seen in #10421.

So, to make this accurate now: I removed all '' in the template, except for the cycler. I then ran a comparisson between the loaded rcParams and the defaultParams.

    rc = mpl.rc_params_from_file(fname,
                                     fail_on_error=True,
                                     use_default_template=False)
   
    d1 = dict(rc.items())
    d2 = dict(mpl.defaultParams.items())
    for k,v in d1.items():
        if d2[k][0] != v:
            print k
            print " ", d2[k][0], v
            print " ", type(d2[k][0]), type(v)

This brought up a number of parameters which did not match. I corrected those which needed to be updated or were obvious to fix. What remains is the following. Here the first item is the one from the defaultParams, the second is the one as read in from the template file. I'm trying to comment on each of them and would be happy about any feedback or action to take.

• keymap.fullscreen
('f', 'ctrl+f') ['f', 'ctrl+f']
<type 'tuple'> <type 'list'>
  All other keymaps seem to stored as list, except this one.

• pgf.preamble
[''] []
<type 'list'> <type 'list'>
  A list with an empty string cannot be created from within the rc file syntax, or can it? If so how?

• path.effects
[] []
<type 'list'> <type 'unicode'>
  How to create an empty list instead of the string '[]'?

• savefig.bbox
standard None
<type 'str'> <type 'NoneType'>
  Despite the file saying savefig.bbox : standard the parsing yields this as None.

• axes.prop_cycle This shows up, simply because two different instances of cycler.Cycler are created.

• backend
Agg agg
<type 'str'> <type 'str'>
  Even if the file says backend : Agg this is parsed as lower case, while the defaultParams have the upper case string stored.

• text.latex.preamble
[''] []
<type 'list'> <type 'list'>
  A list with an empty string cannot be created from within the rc file syntax, or can it? Same as pgf.preamble.

• path.simplify_threshold
0.111111111111 0.111111111111
<type 'float'> <type 'float'>
  I supposed the stored value is 1./9

• svg.hashsalt
None None
<type 'NoneType'> <type 'unicode'>
  String svg.hashsalt : None is not parsed to None, but "None".

If those cases were not showing up, one could ideally add the comparisson above to the tests, making sure not only the keys are correct and the file is correctly read in, but also that the values are up to date and correct.

Partial response: I suspect that an empty list and a list with a single empty string are identical in practice--as they are subsequently used--but this can be checked. If not, it might be that some slight tweaks to the validation functions in rcsetup.py would be in order to fix that, but I would be very cautious about this.
I will be surprised if the matplotlibrc parsing actually requires ugly and confusing things like an isolated comma. The intention was to make it easy to read and write, hence the design that does not use quotes for strings--or didn't, until cycler came along.