Skip to content

Behavior of masked values in colormap plot has changed in 2.2.0 #11039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
arthur00 opened this issue Apr 12, 2018 · 11 comments
Open

Behavior of masked values in colormap plot has changed in 2.2.0 #11039

arthur00 opened this issue Apr 12, 2018 · 11 comments
Labels
keep Items to be ignored by the “Stale” Github Action

Comments

@arthur00
Copy link

Bug report

Bug summary

Something changed the behavior of masked values in colormaps. Before, masked values on a 'jet' colormap would be drawn as white (and could be set to different colors with set_bad). This is no longer the case on matplotlib 2.2.0 - the masked values are being printed out as blue, the same color as valid values, and set_bad has no effect. Verified former behavior to be working on 2.1.1 and 2.0.2.

Not sure this is an issue or expected behavior, but I couldn't find a description of this behavior change anywhere. Please close if this is expected.

Changing interpolation from gaussian to none makes no difference (the sample code was originally used for another bug)

Code for reproduction

import matplotlib.pyplot as plt
import matplotlib.colors

heatmap = [[0] * 32 for _ in range(128)]
cnt = 0
last = 1
for lvl in range(127):
    if lvl % 20 < 10:
        for i in range(32):
            heatmap[lvl][i] = 40
    else:
        for i in range(32):
            heatmap[lvl][i] = 0
jet = plt.get_cmap('jet')
jet.set_bad('white', 1)

plt.imshow(heatmap, origin='lower', interpolation='gaussian', aspect='auto', cmap=jet,
                 norm=matplotlib.colors.LogNorm())
plt.show()

Actual outcome

image

Expected outcome

image

Matplotlib version

  • Operating system: Windows
  • Matplotlib version: 2.2.0
  • Matplotlib backend (print(matplotlib.get_backend())): TkAgg
  • Python version:3.6.3
  • Jupyter version (if applicable):
  • Other libraries:

Installed from pip

@arthur00
Copy link
Author

From #9961, I tried using LogNorm(vmin=1, vmax=5), and I got:

ln(1) -> 0
ln(.5) -> -0.5306 ...
ln(np.nan) -> nan
ln(0) -> masked

But with LogNorm()
ln(1) -> 0
ln(.5) -> 0
ln(np.nan) -> 0
ln(0) -> masked

Numpy version 1.13

@arthur00
Copy link
Author

To recap (some tests were done in #9961): If LogNorm sets vmin != vmax, the graph is plotted correctly:

image

If the vmin == vmax == None (i.e. LogNorm()), the shown effect of a block of blue with no stripes is seen.

@efiring
Copy link
Member

efiring commented Jul 2, 2018

class LogNorm(Normalize):
    """
    Normalize a given value to the 0-1 range on a log scale
    """
    def __call__(self, value, clip=None):
        if clip is None:
            clip = self.clip

        result, is_scalar = self.process_value(value)

        result = np.ma.masked_less_equal(result, 0, copy=False)

        self.autoscale_None(result)
        vmin, vmax = self.vmin, self.vmax
        if vmin > vmax:
            raise ValueError("minvalue must be less than or equal to maxvalue")
        elif vmin <= 0:
            raise ValueError("values must all be positive")
        elif vmin == vmax:
            result.fill(0)

The problem is coming from the last line above, which is throwing away the mask in the special case where there is only a single valid value in the array. I will submit a PR for it.

@efiring
Copy link
Member

efiring commented Jul 2, 2018

Wrong! Investigating...

@efiring
Copy link
Member

efiring commented Jul 2, 2018

The problem is that all the complex rescaling in the resampling code in imshow probably should be skipped for the special case where there is only a single valid value in the array. @tacaswell, sorry, but I think I had better leave the implementation of that special case to you.

@github-actions
Copy link

github-actions bot commented May 4, 2023

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

@github-actions github-actions bot added the status: inactive Marked by the “Stale” Github Action label May 4, 2023
@QuLogic
Copy link
Member

QuLogic commented May 4, 2023

I cannot reproduce this problem with the specified versions. However I can reproduce on main, and a bisect points to the backport of #20511 in 3.4.3. That change does modify LogNorm to remove an accidental modification of the internal data. However, at this point in main, the LogNorm is created out of the LogScale and doesn't have a class-specific autoscale_None that can be overridden. But, having written that change, I still think it is correct.

So in the end, I do have to wonder if imshow is applying the norm parameter correctly.

@QuLogic
Copy link
Member

QuLogic commented May 4, 2023

After some further investigation, I think @efiring was on the right track above. With the refactoring of LogNorm into a LogScale+Normalize derivative, the location of the problem has changed a bit, but it's still about the same:

In [1]: from matplotlib.colors import LogNorm

In [2]: LogNorm(vmin=10, vmax=11)([0, 9, 10, 11, 12, 13])
Out[2]: 
masked_array(data=[--, -1.1054487136015845, 0.0, 1.0, 1.912928473834252, 2.752741260231958],
             mask=[True, False, False, False, False, False],
       fill_value=1e+20)

In [3]: LogNorm(vmin=10, vmax=10)([0, 9, 10, 11, 12, 13])
Out[3]: 
masked_array(data=[0., 0., 0., 0., 0., 0.],
             mask=False,
       fill_value=1e+20)

When vmin != vmax, anything outside produces something outside the 0-1 range expected for a norm and negative numbers for LogNorm are masked. But when they're equal, then only zeros are output, due to:

if self.vmin == self.vmax:
return np.full_like(value, 0)

This can be fixed by something like:

diff --git a/lib/matplotlib/colors.py b/lib/matplotlib/colors.py
index 434bb54235..94a50e1d90 100644
--- a/lib/matplotlib/colors.py
+++ b/lib/matplotlib/colors.py
@@ -1709,8 +1709,6 @@ def _make_norm_from_scale(
                 self.autoscale_None(value)
             if self.vmin > self.vmax:
                 raise ValueError("vmin must be less or equal to vmax")
-            if self.vmin == self.vmax:
-                return np.full_like(value, 0)
             if clip is None:
                 clip = self.clip
             if clip:
@@ -1720,8 +1718,11 @@ def _make_norm_from_scale(
             if not np.isfinite([t_vmin, t_vmax]).all():
                 raise ValueError("Invalid vmin or vmax")
             t_value -= t_vmin
-            t_value /= (t_vmax - t_vmin)
-            t_value = np.ma.masked_invalid(t_value, copy=False)
+            if self.vmin == self.vmax:
+                t_value[t_value > 0] += 1
+            else:
+                t_value /= (t_vmax - t_vmin)
+                t_value = np.ma.masked_invalid(t_value, copy=False)
             return t_value[0] if is_scalar else t_value
 
         def inverse(self, value):

(though I'm not sure if the last line should be in or out of the else.)

However, fixing that does not fix the image. This appears to be due to this extra clipping applied at:

# Clip scaled data around norm if necessary. This is necessary
# for big numbers at the edge of float64's ability to represent
# changes. Applying a norm first would be good, but ruins the
# interpolation of over numbers.
self.norm.autoscale_None(A)
dv = np.float64(self.norm.vmax) - np.float64(self.norm.vmin)
vmid = np.float64(self.norm.vmin) + dv / 2
fact = 1e7 if scaled_dtype == np.float64 else 1e4
newmin = vmid - dv * fact
if newmin < a_min:
newmin = None
else:
a_min = np.float64(newmin)
newmax = vmid + dv * fact
if newmax > a_max:
newmax = None
else:
a_max = np.float64(newmax)
if newmax is not None or newmin is not None:
np.clip(A_scaled, newmin, newmax, out=A_scaled)

Since a LogNorm will autoscale without the 0, it will tell the image code here that vmin==vmax==40, and then the image code clips to a minimum of 40, losing all the zeros before the norm is even called. I have not tested it, but do also wonder if that means that a norm with vmin/vmax within the array limits will not correctly plot under or over colours.

@github-actions github-actions bot removed the status: inactive Marked by the “Stale” Github Action label May 5, 2023
@greglucas
Copy link
Contributor

Is vmin == vmax something we want/need to support? There is ambiguity when calling norm(vmin), should we return 0, 0.5, or 1? It seems like disallowing the case where vmin == vmax would solve a lot of special-casing in other places.

We already apply a nonsingular transform in Colorbar to specifically avoid this case when the norm is on an axis, but I'm wondering if this special-casing should actually be moved upstream to the Normalize class and applied during the autoscaling?

Copy link

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

@github-actions github-actions bot added the status: inactive Marked by the “Stale” Github Action label Jul 12, 2024
@github-actions github-actions bot added the status: closed as inactive Issues closed by the "Stale" Github Action. Please comment on any you think should still be open. label Aug 12, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 12, 2024
@timhoffm
Copy link
Member

The topic of singular norms (vmin == vmax) is not consistently solved. Not a high priority, but we should come up with a consistent approach. See also

@timhoffm timhoffm reopened this Aug 13, 2024
@timhoffm timhoffm added keep Items to be ignored by the “Stale” Github Action and removed status: inactive Marked by the “Stale” Github Action status: closed as inactive Issues closed by the "Stale" Github Action. Please comment on any you think should still be open. labels Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
keep Items to be ignored by the “Stale” Github Action
Projects
None yet
Development

No branches or pull requests

5 participants