Skip to content

FIX: Include 0 when checking lognorm vmin #20488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 24, 2021

Conversation

greglucas
Copy link
Contributor

PR Summary

Change vmin check to less than or equal to 0 rather than strictly less than.

Fixes #20487

PR Checklist

  • Has pytest style unit tests (and pytest passes).
  • Is Flake 8 compliant (run flake8 on changed files to check).
  • New features are documented, with examples if plot related.
  • Documentation is sphinx and numpydoc compliant (the docs should build without error).
  • Conforms to Matplotlib style conventions (install flake8-docstrings and run flake8 --docstring-convention=all).
  • New features have an entry in doc/users/next_whats_new/ (follow instructions in README.rst there).
  • API changes documented in doc/api/next_api_changes/ (follow instructions in README.rst there).

@@ -533,7 +533,7 @@ def _make_image(self, A, in_bbox, out_bbox, clip_bbox, magnification=1.0,
# that may have moved input values in/out of range
s_vmin, s_vmax = vrange
if isinstance(self.norm, mcolors.LogNorm):
if s_vmin < 0:
if s_vmin <= 0:
s_vmin = max(s_vmin, np.finfo(scaled_dtype).eps)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is there a max here? .eps should be greater than s_vmin due to the condition, so no need for max, or else drop the if instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with that.

@jklymak
Copy link
Member

jklymak commented Jun 23, 2021

Yeah I don't think this fix is correct. I debugged for a while and self.vmin changes from 100 to 0 in the offending test. I have zero understanding of where that may have happened, but I think this is hiding the problem.

@QuLogic
Copy link
Member

QuLogic commented Jun 23, 2021

s_vmin and s_vmax are what temporarily change norm.vmin and norm.vmax. That code came in at the same time as the test, in #18458. The intention is to re-scale things to not go out of bound somewhere or other, but this block ensures that we don't rescale too much and go out-of-bounds for log.

The difference seems to be with A in _make_image, at least by line 396 or so, but I think it's not modified before that. In NumPy 1.17.5, the array contains 1e20, and -1, but in 1.21, the array is masked. After scaling, the -1 is still -1, so it easily matches the condition before the change above, and is replaced by epsilon. When it's masked, the min and max of the array are 1e20, and eventually that works out to 0 at the end of the re-scaling stuff. I haven't worked out why the mask is added/dropped in one version or the other.

@greglucas
Copy link
Contributor Author

I ran this through a debugger and A has the negative values masked in the new version of Numpy, whereas before the mask was left off (a_min is -1 previously, 1e20 now). I think the offending line is:

super().autoscale_None(np.ma.masked_less_equal(A, 0, copy=False))

Maybe due to some in-place editing of the array now within the autoscaling.

I'll try and look more into this later today.

@greglucas
Copy link
Contributor Author

I think this PR fixed the old behavior where the MaskedArray wasn't being modified even with a copy=False: numpy/numpy#18967

We can fix this test by creating a copy, which is the default in numpy. Alternatively, do we want the autoscaling function to modify the passed in array and mask elements or do we want autoscale to only set the vmin/vmax and never modify the input array? I can see both sides...

@tacaswell
Copy link
Member

✔ 15:56:17 $ git diff
diff --git a/lib/matplotlib/colors.py b/lib/matplotlib/colors.py
index e0c42c5b69..5e9c6f6dbd 100644
--- a/lib/matplotlib/colors.py
+++ b/lib/matplotlib/colors.py
@@ -1545,11 +1545,11 @@ class LogNorm(Normalize):
 
     def autoscale(self, A):
         # docstring inherited.
-        super().autoscale(np.ma.masked_less_equal(A, 0, copy=False))
+        super().autoscale(np.ma.masked_less_equal(np.ma.getdata(A), 0))
 
     def autoscale_None(self, A):
         # docstring inherited.
-        super().autoscale_None(np.ma.masked_less_equal(A, 0, copy=False))
+        super().autoscale_None(np.ma.masked_less_equal(np.ma.getdata(A), 0))
 
 
 @_make_norm_from_scale(

This patch fixes the tests

@QuLogic
Copy link
Member

QuLogic commented Jun 23, 2021

That code appears to have not 'made a copy' for 11 years: 3101e00 I think the intention was don't copy if-not-needed (and not as the NumPy docs intended), but it's hard to say now. It's also possible that our input changed from maybe-masked to always-masked with the changes to how input data is (not/now) copied, and that had this side effect we didn't expect.

@greglucas
Copy link
Contributor Author

That patch will remove already masked values that were present though and it removes the copy, so I'd suggest just removing the copy=False instead of adding the getdata call.

@tacaswell
Copy link
Member

🤦🏻 meant to leave the copy...

diff --git a/lib/matplotlib/colors.py b/lib/matplotlib/colors.py
index e0c42c5b69..3766e57d03 100644
--- a/lib/matplotlib/colors.py
+++ b/lib/matplotlib/colors.py
@@ -1545,11 +1545,19 @@ class LogNorm(Normalize):
 
     def autoscale(self, A):
         # docstring inherited.
-        super().autoscale(np.ma.masked_less_equal(A, 0, copy=False))
+        super().autoscale(
+            np.ma.masked_less_equal(
+                np.ma.getdata(A), 0, copy=False
+            )
+        )
 
     def autoscale_None(self, A):
         # docstring inherited.
-        super().autoscale_None(np.ma.masked_less_equal(A, 0, copy=False))
+        super().autoscale_None(
+            np.ma.masked_less_equal(
+                np.ma.getdata(A), 0, copy=False
+            )
+        )
 
 
 @_make_norm_from_scale(

also passes. I think we want to try to avoid making needless (internal) copies of image data which can be big.

@tacaswell
Copy link
Member

And I think we still need a version of this PR to fix a different bug where the vmin of the norm is very far from the minimum of the data.

On the phone @QuLogic convinced me that @greglucas is right and we want to a) not copy the underlying data b) preserve any existing mask the use gave us c) enrich that mask with where the data is less than 0.

@jklymak
Copy link
Member

jklymak commented Jun 23, 2021

This seems like a good bit of dancing around just so we can call super()?

@jklymak
Copy link
Member

jklymak commented Jun 23, 2021

Sorry, I should elaborate. I don't think you want to permanently add to the users mask, because you can't go back to a different norm at that point.

@tacaswell
Copy link
Member

We are still not messing with the users copy (we make a copy earlier in the process), but with numpy 1.21.0 we are mutating our internal copy of the mask.

@jklymak
Copy link
Member

jklymak commented Jun 23, 2021

But we don't re-supply ourselves with the user's copy, so we would be blanking out all negative data if we change the norm on the scalarmappable with set_norm?

@greglucas greglucas force-pushed the fix-image-lognorm branch from 6b3738e to 4760953 Compare June 23, 2021 21:48
@greglucas
Copy link
Contributor Author

OK, I think we should leave the autoscaling copy=False alone and use it as it currently is to preserve the passed-in masks and not copy anything as mentioned in comments above.

The issue was that before our s_vmin was being set to -1 because the array was populated with -1's. If we had populated the array with 0's we would have been in the same boat as now.

I added another test to duplicate this behavior, except using a full range of 1 -> 1e20 by setting the array to 1's initially to verify the actual large range case gets covered too.

Copy link
Member

@QuLogic QuLogic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should do something about the copying change, but that just exposed a different bug here that needs to be fixed anyway.


ax = fig_ref.subplots()
im = ax.imshow(data, norm=colors.Normalize(vmin=1, vmax=data.max()),
interpolation='nearest', cmap='viridis')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure you don't need to actually duplicate the test function, just use @pytest.mark.parametrized(). But that is a minor nitpick, and my earlier approval still stands. This does fix an actual bug, even if it isn't the same bug everyone thought it was going to fix.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great suggestion, thanks!

Change vmin check to less than or equal to 0 rather than strictly
less than.
@greglucas greglucas force-pushed the fix-image-lognorm branch from 4760953 to 33f3526 Compare June 24, 2021 01:48
@greglucas
Copy link
Contributor Author

I'm not following what the other bug with copying is actually? I think the copy=False is doing what we think it is supposed to do now, it is just this test got unlucky because of < catching the negative values before that aren't there anymore.

@QuLogic
Copy link
Member

QuLogic commented Jun 24, 2021

I'll open a PR for that soon.

@QuLogic QuLogic merged commit c6c85e6 into matplotlib:master Jun 24, 2021
meeseeksmachine pushed a commit to meeseeksmachine/matplotlib that referenced this pull request Jun 24, 2021
timhoffm added a commit that referenced this pull request Jun 24, 2021
…488-on-v3.4.x

Backport PR #20488 on branch v3.4.x (FIX: Include 0 when checking lognorm vmin)
@greglucas greglucas deleted the fix-image-lognorm branch June 24, 2021 13:55
@ianhi ianhi mentioned this pull request Jun 24, 2021
5 tasks
QuLogic added a commit to fedora-python/matplotlib that referenced this pull request Aug 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test_huge_range_log is failing...
5 participants