[NF] Add 'truncate' and 'join' methods to colormaps. #7716

lkilcher · 2016-12-31T18:09:00Z

This is functionality that I've been wanting built-in for a while. The inspiration largely comes from here and here. I know @tacaswell had asked that this functionality be included way back in 2014.

I think there is still an 'issue' here if you have/want a discontinuous LinearSegmentedColormap because this implementation just smooths those over. Still, if N is high, that shouldn't be too much of a problem. I've developed a version of the LinearSegmentedColormap.truncate method that actually crops the cdict appropriately, but since MEP21 seems to indicate that MPL may be moving away from LSC's, I didn't bother including it. Plus, it's a bit tricky to create a join method that correctly joins LSC's to listed cmaps.

Suggestions welcome!

dstansby · 2016-12-31T18:44:48Z

lib/matplotlib/colors.py

+            The fraction of the new colormap that should be occupied
+            by self. By default, this is ``self.N / (self.N +
+            other.N)``.
+


What does the N kwarg do?

OK. I've updated the docstring.

codecov-io · 2016-12-31T20:10:46Z

Current coverage is 62.10% (diff: 100%)

Merging #7716 into master will decrease coverage by 4.48%

@@             master      #7716   diff @@
==========================================
  Files           109        174     +65   
  Lines         46648      56062   +9414   
  Methods           0          0           
  Messages          0          0           
  Branches          0          0           
==========================================
+ Hits          31061      34815   +3754   
- Misses        15587      21247   +5660   
  Partials          0          0

Powered by Codecov. Last update 07f6427...615ca22

tacaswell · 2016-12-31T20:40:40Z

lib/matplotlib/colors.py

+        cmap_trunc = cmap.truncate(0.2, 0.7)
+
+        """
+        assert minval < maxval, "minval must be less than maxval"


Can you use if not cond: raise instead of asserts? A while ago we went through and removed all of the run-time uses of asserts for input validation as (in principle) the interpreter can decide to ignore the assert statements.

tacaswell · 2016-12-31T20:42:16Z

lib/matplotlib/colors.py

@@ -742,6 +742,107 @@ def func_r(x):

        return LinearSegmentedColormap(name, data_r, self.N, self._gamma)

+    def join(self, other, name=None, frac_self=None, N=None):


It might be a good idea to also add add a __add__ method so you can do cm1 + cm2 and get a 50/50 map? Could be too cute for it's own good.

I had this idea as well, but thought reviewers would find it 'too cute'! It's included now.

tacaswell · 2016-12-31T20:56:58Z

lib/matplotlib/colors.py

@@ -742,6 +742,107 @@ def func_r(x):

        return LinearSegmentedColormap(name, data_r, self.N, self._gamma)

+    def join(self, other, name=None, frac_self=None, N=None):


Maybe put a generic version of this (using ListedColormap in all case on Colormap? Might also make sense to have a generic join_color_map(*cmaps, fractions=None) as a top-level function (rather than as methods)?

tacaswell · 2016-12-31T21:13:26Z

👍 , left some comments about the API

This will definitely need a whats_new entry.

lkilcher · 2017-01-01T17:45:52Z

Most of the suggested changes have been made. A few things:

Does the whats_new entry go into doc/api/api_changes.rst?
Any suggestions on where to put the join_colormaps function?
I had to remake the image tests when I switched to returning only ListedColormap. We probably don't need the lsc test images anymore. Agree?

lkilcher · 2017-01-01T19:42:41Z

After adding the __add__ method, it got me thinking about how to do something similar for truncate. Since these ideas are a bit half-baked, I started a new branch. Thoughts on this?

lkilcher · 2017-01-30T23:03:29Z

@tacaswell I'm sure you're busy. I just wanted to make sure this wasn't forgotten. Any thoughts on my questions above?

WeatherGod · 2017-01-31T18:13:33Z

That idea is interesting. I would ditch the args-style indexing (e.g., `new_cm = cmap[0.2, 0.7, 64]`). But the slicing approach really makes sense, and to also allow the numpy-style complex slicing is quite clever.

…

On Mon, Jan 30, 2017 at 6:03 PM, Levi Kilcher ***@***.***> wrote: @tacaswell <https://github.com/tacaswell> I'm sure you're busy. I just wanted to make sure this wasn't forgotten. Any thoughts on my questions above? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7716 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-EBi8f1Uy45wvMfPOhWmn-l9l7_hks5rXmxCgaJpZM4LYmOg> .

lkilcher · 2017-02-02T17:43:59Z

Can anyone help me understand what is breaking here?

dstansby · 2017-02-02T17:46:31Z

It looks like it's just PEP8 violations, for lines that are longer than 79 characters. You should be able to see which lines are causing the error here: https://travis-ci.org/matplotlib/matplotlib/jobs/197464743#L1767

lkilcher · 2017-08-16T20:01:12Z

Not sure if there is bandwidth to include this in 2.1, but I'm available to work on it. I think I've addressed all existing comments, except for the whats_new entry. Where do I put that?

Also, I've addressed @WeatherGod 's comments on the proposed __getitem__ method. I'll plan to merge that into this soon, unless there are objections?

jklymak

This looks interesting and helpful. Needs to pass the tests. Not sure if I think we need an image test.

jklymak · 2018-05-17T16:15:49Z

lib/matplotlib/tests/test_colorbar.py

+    plt.pcolormesh(data, cmap=cmap)
+    plt.colorbar(orientation='vertical')
+
+


Not sure this requires image comparisons (which are repo-heavy). Can you just test on the returned colormaps?

Hi Jody! I hadn't thought of this, but it makes sense. Are the colormaps sitting somewhere that I can compare to, or... ?

Hi Levi, I'd just evaluate the colormap at 5 or 6 locations and assert that the values returned are what you say they should be. That seems trivial but if someone mucks with your code they will have to also change the test and thats a good warning to double check what was done.

This is based largely on https://gist.github.com/denis-bz/8052855 Which was based on http://stackoverflow.com/a/18926541/2121597

Note that these methods now always return `ListedColormap`. Also: switch from `assert` to `raise ValueError`.

reorder `join` inputs (for consistency) +doc fixes/changes

lkilcher · 2018-05-21T16:14:58Z

OK. I think this is ready to go. Let me know if there is anything else you need.

Very stale

jklymak

This looks really useful to me. Not 100% sure about the format of the getitem string, and I'm not even sure where this would render in the docs. I also wish this had an actual example somewhere so folks will know about it, otherwise, I think the only documentation is in the whats-new. OTOH that could be done for a second PR...

jklymak · 2018-08-18T21:23:54Z

lib/matplotlib/colors.py

+
+        # ### float indexing
+        # for float-style indexing, the values must be in [0.0, 1.0]
+        # Truncate the colormap between 20 and 80%.


Typo? 80->60?

jklymak · 2018-08-20T20:19:30Z

At the weekly call this week this got substantial discussion. The upshot was that this seems a lot of maintenence, support, and API overhead for something that the user should be able to do themselves, perhaps with better documentation. Basically truncating and subsetting colormaps boils down to making the appropriate matrix to feed to ListedColormap via whatever tools the user is comfortable using in numpy to achieve that end. The major disadvantage of doing it the proposed way, was felt to be the ambiguity of colormap length, and hence the diffculting of knowing whether the join point of two concatenated colormaps was in the middle, or at 1/3 etc. If people have to explicitly specify the length of the colormap, they are most of the way to just manipulating the arrays directly.

The Todo here is to add a section to https://matplotlib.org/tutorials/colors/colormaps.html#sphx-glr-tutorials-colors-colormaps-py that explains colormap manipulation better, and perhaps adding an example or two to the example directory.

@lkilcher thanks so much for the contribution, and sorry it languished so long. If you want to give input to the examples when those PRs come in, I'll ping you, or contributions along those lines would be most welcome.

lkilcher · 2018-08-21T22:06:30Z

This is disappointing, and it's too bad this conversation didn't happen before I put time into the tests etc.. I can understand that __getitem__ is more than the maintainers wanted to handle, but .join and truncate are so necessary! God knows how much time I've spent digging into the code to 're-remember' how to join/truncate colormaps (and it's been changing)! If people are worried about N ambiguity, make that a required argument of the functions.

Making a clearly documented interface to the arrays makes sense, and certainly replaces the need for __getitem__, but I still think cm = cm1 + cm2 -- or at least cm = join_colormaps(cm1, cm2) -- is a lot nicer than cm = ListedColormap(np.vstack(cm1.array, cm2.array)).

I apologize for coming across as a disgruntled contributor. I just know quite a few people who find MPL colormaps very cumbersome, and I put time into this because it got positive feedback from some senior MPL devs. I also know there are those out there who think creating/modifying new colormaps is a bad idea, but I find that reasoning arrogant. Hopefully someone will actually work on MEP21 (especially killing LSCs), and perhaps a simple join_colormaps func (and __add__ method) will make sense at that time.

I'll track of your progress on #11905 and see if I have anything to add.

jklymak · 2018-08-21T22:28:00Z

FWIW I share your concern about the opacity of colormaps and how to create them, hence #11905, because info about ListedColormaps is completely missing from the existent docs. Frankly, I didn't know the class existed before Monday. Given that class, however, we can go back and forth relatively painlessly between np arrays and colormaps, it seemed worth trying to improve the documentation first. If the consensus is still that its still too clunky to manipulate colormaps, then we can revisit the idea of allowing the more holistic solution proposed here.

WRT being able to create your own colormaps, I hope I didn't make it sound like people shouldn't be allowed to do that, or that anyone on the dev team thinks that.

In general, the devs are concerned about the balance between "what we can do" and "what we should do". Some features got out of control in the past, and are now maintenance drags on the library. I think the current philosophy is to try to keep things as lightweight as possible and to have fewer black boxes.

jklymak · 2018-09-28T04:55:01Z

Closing in lieu of documentation of how to do the same thing w/ ListedColormaps. @lkilcher again apologies for the long review process that eventually didn't get merged. We really want people to contribute, so we try to avoid this, but sometimes ideas fall between the cracks.

lkilcher force-pushed the colormap_utils branch from c70c1b5 to 615ca22 Compare December 31, 2016 18:40

dstansby reviewed Dec 31, 2016

View reviewed changes

tacaswell added this to the 2.1 (next point release) milestone Dec 31, 2016

tacaswell previously requested changes Dec 31, 2016

View reviewed changes

lkilcher force-pushed the colormap_utils branch from 688ff01 to a0ee437 Compare February 1, 2017 22:22

tacaswell modified the milestones: 2.1 (next point release), 2.2 (next next feature release) Aug 29, 2017

lkilcher force-pushed the colormap_utils branch from 7680655 to 5ac5bb0 Compare September 27, 2017 19:36

jklymak reviewed May 17, 2018

View reviewed changes

jklymak modified the milestones: needs sorting, v3.0 May 17, 2018

jklymak added the status: needs revision label May 17, 2018

lkilcher added 7 commits May 17, 2018 16:33

NF - Add 'truncate' and 'join' methods to colormaps.

ea42661

This is based largely on https://gist.github.com/denis-bz/8052855 Which was based on http://stackoverflow.com/a/18926541/2121597

Some pep8 fixes.

e15203a

Update .join docstring to include the parameter.

42bdcb0

Move methods to Colorbar.

60ebf94

Note that these methods now always return `ListedColormap`. Also: switch from `assert` to `raise ValueError`.

Add the __add__ method, and doc.

f06bf46

Add a name parameter to truncate

81ddd1c

reorder `join` inputs (for consistency) +doc fixes/changes

Refactor onto a 'join_colormaps' function

71742d7

lkilcher and others added 7 commits May 17, 2018 16:33

Some ideas for a colormap __getitem__ method.

fa9f56a

Fix some floating-point arithmetic issues for low-N.

aba18e0

Simplify getitem.

652b4cf

Add more tests.

c2bd712

pep8

6f0a342

Add whats_new and update docstring

9de0239

Compare cmap values in tests (not image comparisons)

48172c7

lkilcher force-pushed the colormap_utils branch from 65becd1 to 48172c7 Compare May 18, 2018 20:31

lkilcher and others added 3 commits May 18, 2018 14:54

pep8 fixes

8592695

Increase coverage.

17e576e

Test: 100% coverage!

9caba24

jklymak self-assigned this May 28, 2018

jklymak added status: needs review and removed status: needs revision labels May 28, 2018

tacaswell modified the milestones: v3.0, v3.1 Jul 9, 2018

Merge branch 'master' into colormap_utils

2384d0b

jklymak approved these changes Aug 18, 2018

View reviewed changes

jklymak changed the title ~~[WIP] [NF] Add 'truncate' and 'join' methods to colormaps.~~ [NF] Add 'truncate' and 'join' methods to colormaps. Aug 19, 2018

jklymak added Documentation and removed status: needs review labels Aug 20, 2018

jklymak mentioned this pull request Aug 20, 2018

DOC: Colormap tutorial needs info on manipulating colormaps #11903

Closed

jklymak closed this Sep 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NF] Add 'truncate' and 'join' methods to colormaps. #7716

[NF] Add 'truncate' and 'join' methods to colormaps. #7716

lkilcher commented Dec 31, 2016

dstansby Dec 31, 2016

lkilcher Dec 31, 2016

codecov-io commented Dec 31, 2016 •

edited

Loading

tacaswell Dec 31, 2016

tacaswell Dec 31, 2016

lkilcher Jan 1, 2017

tacaswell Dec 31, 2016

tacaswell commented Dec 31, 2016

lkilcher commented Jan 1, 2017

lkilcher commented Jan 1, 2017 •

edited

Loading

lkilcher commented Jan 30, 2017

WeatherGod commented Jan 31, 2017 via email

lkilcher commented Feb 2, 2017

dstansby commented Feb 2, 2017

lkilcher commented Aug 16, 2017

jklymak left a comment

jklymak May 17, 2018

lkilcher May 17, 2018

jklymak May 17, 2018

lkilcher commented May 21, 2018

jklymak left a comment

jklymak Aug 18, 2018

jklymak commented Aug 20, 2018

lkilcher commented Aug 21, 2018

jklymak commented Aug 21, 2018

jklymak commented Sep 28, 2018

		@@ -742,6 +742,107 @@ def func_r(x):

		return LinearSegmentedColormap(name, data_r, self.N, self._gamma)

		def join(self, other, name=None, frac_self=None, N=None):

		plt.pcolormesh(data, cmap=cmap)
		plt.colorbar(orientation='vertical')

[NF] Add 'truncate' and 'join' methods to colormaps. #7716

[NF] Add 'truncate' and 'join' methods to colormaps. #7716

Conversation

lkilcher commented Dec 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Dec 31, 2016 • edited Loading

Current coverage is 62.10% (diff: 100%)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tacaswell commented Dec 31, 2016

lkilcher commented Jan 1, 2017

lkilcher commented Jan 1, 2017 • edited Loading

lkilcher commented Jan 30, 2017

WeatherGod commented Jan 31, 2017 via email

lkilcher commented Feb 2, 2017

dstansby commented Feb 2, 2017

lkilcher commented Aug 16, 2017

jklymak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lkilcher commented May 21, 2018

jklymak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jklymak commented Aug 20, 2018

lkilcher commented Aug 21, 2018

jklymak commented Aug 21, 2018

jklymak commented Sep 28, 2018

codecov-io commented Dec 31, 2016 •

edited

Loading

lkilcher commented Jan 1, 2017 •

edited

Loading