A quick question: what are we supposed to do with non-masked arrays containing NaNs? Normalize will return a NaN:

>>> import matplotlib.colors as mcolors
>>> n = mcolors.Normalize(0, 2)
>>> n([1, np.NaN])
masked_array(data = [ 0.5  nan],
             mask = False,
       fill_value = 1e+20)

Of course, the current version of BoundaryNorm will throw annoying warnings. Am I supposed to mask NaNs automatically?

I thought ma.asarray masks them automatically?

On Thu, Jul 30, 2015 at 12:10 PM, Fabien Maussion notifications@github.com
wrote:

A quick question: what are we supposed to do with non-masked arrays
containing NaNs? Normalize will return a NaN:

import matplotlib.colors as mcolors>>> n = mcolors.Normalize(0, 2)>>> n([1, np.NaN])
masked_array(data = [ 0.5 nan],
mask = False,
fill_value = 1e+20)

Of course, the current version of BoundaryNorm will throw annoying
warnings. Am I supposed to mask NaNs automatically?

—
Reply to this email directly or view it on GitHub
#4824 (comment)
.

Nope. However, fix_invalid seems to do the job:

>>> x = [1., -1, np.nan, np.inf]
>>> np.ma.asarray(x)
masked_array(data = [  1.  -1.  nan  inf],
             mask = False,
       fill_value = 1e+20)

>>> np.ma.fix_invalid(x)
masked_array(data = [1.0 -1.0 -- --],
             mask = [False False  True  True],
       fill_value = 1e+20)

Sorry to be so picky, but if we expect the following behaviour with a valid scalar:

>>> import matplotlib.colors as mcolors
>>> bn = mcolors.BoundaryNorm([1,2,3], 2)
>>> bn(1.5)
0

what should we expect from the following call?

>>> bn(np.NaN)
???

On my working version this is what I get:

>>> bn(np.NaN)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mowglie/Documents/git/matplotlib/lib/matplotlib/colors.py", line 1291, in __call__
    ret = int(ret[0])  # assume python scalar
  File "/home/mowglie/.pyvirtualenvs/mpl_devel/lib/python3.4/site-packages/numpy/ma/core.py", line 3910, in __int__
    raise MaskError('Cannot convert masked element to a Python int.')
numpy.ma.core.MaskError: Cannot convert masked element to a Python int.

Nans are presently not supported in the Normalize API. Obviously, this could be changed. It would impose a performance penalty for large arrays. Overall, this is an example of how the mpl API has evolved, without a clear statement of where operations like checking for invalid values should be done, and where they should not. If we do the checking at every level of the API, the performance penalties mount up. Certainly we want to do it at the highest level, like plot and contourf. But Normalize is at a lower level, so we could leave it as-is and just say it is the caller's responsibility to clean up the input.

Whenever it is decided that we need np.masked_invalid or np.fixed_invalid (which I think is a little more general, but slower), it would be good to use the copy kwarg to avoid a copy unless it is really needed. With fixed_invalid it is probably needed in many cases, because fixed_invalid fills the invalid values. Masked_invalid just returns a masked array without changing any underlying data values.

I see what you mean. I think we shouldn't care about users doing something like BoundaryNorm(np.NaN). The current error message is clear enough.

On my working version I make a copy of the input: _x = ma.fix_invalid(x) but I am ok with changing it to: x = ma.masked_invalid(x). In that case we will change the user's input to a masked array without changing the value. Is that OK to modify user's input that way?

Even masked_invalid has a copy=True default, so to avoid a copy it
needs to be set to False. In that case we are still not changing the
user's input at all; we are just giving ourselves an object containing
the original user's data together with a mask, and methods that know
what to do with the mask. So long as we don't modify the data values,
there are no side effects.
My larger question, however, is whether any of the Normalize family
are at a level where they should be checking for nan, given that we
do this at the higher API level, and don't want to keep doing it at
each level down the line. Attn: @WeatherGod, @mdboom, @tacaswell.

Is it common practice in Matplotlib to modify user input? Duck arrays such as Pandas TimeSeries and Xray DataArray will be silently turned into a masked_array...

Yes, I understand now. I actually come from IDL, where def f(x): x = x+1 would actually change the value of x. I tend to forgot that in python this is not the case. I just committed a new version with a few more tests for masked arrays and clipping, tell me what you think.

Sorry about the PEP error. Next time I'll check it prior to the commit. I just opened an issue about a possible enhancement, tell me what you think about it: #4850

I corrected the PEP error (hopefully) and replaced the call to ma.masked_invalid with a call to self.process_value in order to be consistent with all other Normalize classes. Therefore if we want to change the general behaviour to do more checks, a change to process_value will suffice.

BoundaryNorm is different from all other classes in that in returns integers and not floats. It will produce warnings if run on non-masked NaN values, but I think it's ok like this.

@efiring, @tacaswell should I do anything else for the PR to be merged/closed? I am away from my mails for the next week but I gess it's not that urgent anyway.

Thanks very much!