Skip to content

MultivarColormap and BivarColormap #28454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Aug 23, 2024

Conversation

trygvrad
Copy link
Contributor

@trygvrad trygvrad commented Jun 25, 2024

Creation of bivariate and multivariate colormap classes (colors.BivarColormap, colors.MultivarColormap)

PR summary

This PR has been in development during 2024 GSOC as a responese to #14168 Feature request: Bivariate colormapping.
This requires the introduction of multiple new classes, and changes to different parts of the codebase. It has therefore been suggested to introduce this functionality over multiple PRs, so that the changes are easier to review one of several PRs that will resolve #14168. See also PR #28428

The end goal of this is to allow for the following funcitonality:

fig, axes = plt.subplots(1, 2, figsize = (12,4))
pm = axes[0].pcolormesh((A,B,C), cmap = '3VarSubA', vmax = (0.4, 0.6, 0.5))
cb0, cb1, cb2 = fig.colorbars(pm, shape = (-1,2))
axes[0].set_title('Subtractive multivariate colormap')
pm = axes[1].pcolormesh((A,B,C), cmap = '3VarAddA', vmax = (0.4, 0.6, 0.5))
cb0, cb1, cb2 = fig.colorbars(pm, shape = (-1,2))
axes[1].set_title('Additive multivariate colormap')

image

fig, axes = plt.subplots(1, 2, figsize = (12,4))

pm = axes[0].pcolormesh((im0,im1), cmap = 'BiOrangeBlue', vmin = -1, vmax = 1)
cax = fig.colorbar_2D(pm, shape = (-1,2))
axes[0].set_title('Square 2d colormap')

pm = axes[1].pcolormesh((im0,im1), cmap = 'BiCone', vmin = -1, vmax = 1)
cax = fig.colorbar_2D(pm, shape = (-1,2))
axes[1].set_title('Circular 2d colormap')

image

The current class hierarchy can be visualized as follows:
image

This PR introduces classes for bivariate (2D) and multivariate (ND) colormaps, as shown in the figure below. It also includes tests for the new types of colormaps.

rect484

This PR does not contain the class cm.Vectormappable, nor does it contain the changes needed in the plotting functions, tests etc. These will follow in separate PRs, as outlined in the figure above. (a full implementation is available in the branch here)

PR checklist

¹This is the first of a series of PRs, and cannot by itself close #14168.

Copy link
Member

@story645 story645 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haven't wrapped my head yet around what the call functions are doing

"""Get the color for masked values."""
return np.array(self._rgba_bad)

def set_bad(self, color='k', alpha=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to make the new colormaps immuatble. This should have been the case with current colormaps as well. Instead of modifying the existing colormap, return a new colormap with modified properties; c.f. Colormap.with_extremes, #14645 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is reasonable suggestion, but there is a practical complication.
MultivarColormap contains a list of Colormap objects, which themselves are not immutable.
(I implemented MultivarColormap.__getitem__() as a way to get the individual constituent colormaps, because I believe this is useful functionality.)
If we make MultivarColormap immutable while the constituent colormaps are not I'm afraid we will run into some trouble in this regard.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this is not optimal. But it's not as bad as one may think. Since colormaps are not immuatable colormaps[name] or get_cmap(name) return a new copy to prevent users from modifying the builtin colormaps. This also means, if you provide these colormaps as input to MultivarColormap and don't keep references, you're safe. Additionally, I find it important from an education perspective to not give the users the tools to modify existing colormaps, even if we cannot ensure full immutability. And finally, we may be able to move towards immutable colormaps one day and thus should not put additional obstacles in our way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just so we are clear, this is the kind of use case I am envisioning:

cmaps = matplotlib.multivar_colormaps['2VarAddA']
norm_growth = matplotlib.colors.Normalize(vmin=2, vmax=10)
norm_capita = matplotlib.colors.Normalize(vmin=50, vmax=110)

p0 = matplotlib.collections.PatchCollection(patches, cmap=cmaps[0], norm=norm_capita)
p1 = matplotlib.collections.PatchCollection(patches, cmap=cmaps[1], norm=norm_growth)
p2 = matplotlib.collections.PatchCollection(patches, cmap=cmaps,  norm=(norm_capita, norm_growth))
...

image

Where two datasets are shown independently, and also together, so that correlations can (ideally) be more easily identified. In order to use the same colormap both to visualize data independently, and together the constituent Colormap objects of the MultivarColormap object needs to accessed, i.e. via:

    def __getitem__(self, item):
        return self.colormaps[item]

Are you thinking we should change this to:

    def __getitem__(self, item):
        return self._colormaps[item].copy()

Where the member variable has been made private and we only return a copy.
With these changes we can then make the MultivarColormap objects immutable.

(PS: I would like to use the above figure as part of an example in a later PR)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not exactly what I meant. I'm merely suggesting not to add any in-place modifying functions.

The copy comments were more to explain what happens with colormaps currently, and that you be don't have to be too concerned that somebody changes the input s you use under you. I haven't thought about extracting colormaps from the multivar colormaps. In particular here, I'd not make a copy because the colormaps are logically linked.

On a general note, I find this example difficult to follow; e.g. Washington is light grey. That this is a combination of saturated blue and saturated brown is hard to realize. This multivar approach has the fundamental problem that it can produce colors that are far from participating colormaps. The degree of the problem depends on the data and on the chosen colormaps. I feel for this visualization a bivar mapping like image is more suited. Here one can easily see: black=both low, white=both high, orange/blue=one high and one low.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This discussion got a bit sidetracked.
Regarding the original question, I removed all set_ functions and introuduced with_extremes() for both BivarColormap and MultivarColormap.

@story645 story645 added this to the v3.10.0 milestone Jul 3, 2024
@trygvrad trygvrad force-pushed the Multivariate-colormap-types branch from 365df72 to d11d47c Compare July 9, 2024 12:50
@JBorrow
Copy link

JBorrow commented Jul 11, 2024

These look great! I'll take a closer look soon, but here's some info that I talked with folks at SciPy2024 about:

@trygvrad
Copy link
Contributor Author

trygvrad commented Jul 11, 2024

These look great! I'll take a closer look soon, but here's some info that I talked with folks at SciPy2024 about:

😄

I think we try to save the discussion on specific colormaps for a later PR, but the Millennium colormap sounds to me like the kind of thing to include.

For now, there is colors.BivarColormapFromImage(name, np.array) that lets you make colormaps from numpy arrays of shape (n,m,3) or (n,m,4). Based on this, it is easy to make a function that reads from an image file, and it becomes more a question of what the public API should look like. I'll try to keep it in mind and get back to it when it in a later PR, as I have tried to keep this PR limited to just the new colormap classes. If we need additional ways to access them we can easily add them later, for example when working on tutorials/examples.

@JBorrow
Copy link

JBorrow commented Jul 16, 2024

This is a really nice PR. In addition to the above, I think the only thing that I would say should be super clarified down the line is exactly how two 'clashing' 1D colour maps can be smashed together to always produce in-range values. For instance A maps to [0, 0, 199] and B maps to [0, 0, 205] leading to [0, 0, 404] in 'Add' mode. I'm a little concerned that the existing implementation here can lead to low-contrast and low-brightness images (instead of using brightness-preserving blend modes).

@trygvrad
Copy link
Contributor Author

This is a really nice PR. In addition to the above, I think the only thing that I would say should be super clarified down the line is exactly how two 'clashing' 1D colour maps can be smashed together to always produce in-range values. For instance A maps to [0, 0, 199] and B maps to [0, 0, 205] leading to [0, 0, 404] in 'Add' mode. I'm a little concerned that the existing implementation here can lead to low-contrast and low-brightness images (instead of using brightness-preserving blend modes).

This is certainly an issue 😅, but it quickly becomes complicated. While it is perhaps not ideal to simply truncate the colors, it is not clear to me what the alternative is.

That said, if the colormaps are designed with this in mind, it becomes less of an issue.
Consider the following colormaps:
image
(Example data from: https://arxiv.org/abs/1812.10366)
You will notice that the this combination of colormaps never leaves sRGB space, thus there is no issue. However, as you say, the color palette is perhaps a bit drab (low in saturation).
This colormap is designed with the following design criteria:

  1. Each component by itself is perceptually uniform
  2. Each component has similar saturation and lightness
  3. Equal mixing of the components provides a grayscale
  4. No combination leaves sRGB space.

In perceptually linear colorspace (CAM02-LCD) the "rhombohedron" formed by this colormap looks like this:
image
(3D view of colorspace)

I find that if we want to have colormaps with greater saturation, we have to compromise on the points above and/or represent colors in a different colorspace. I wold really like to do the color mixing in CAM02-LCD directly, but the transformation $sRGB\leftrightarrow CAM02-LCD$ is both complicated and numerically unstable. OkLab¹ is an alternative, but last time I tried I found that it was not suitable for mixing colors around the extremes (black/white).

Alternatively, if we are willing to compromise on point 4. (No combination leaves sRGB space) above, we can get higher saturation:

image
(3D view of colorspace)

If the data is sparse (i.e. each image is mostly black and the different images are bright in different areas), as is the case in the image above, this tradeoff might be worth it. Fluorescent microscopy (shown above) and element maps from X-ray fluorescence are the two main applications I am aware of where color mixing of 3+ channels is routine, and both of these techniques frequently produce sparse data. (Let me know if you know of another domain frequently uses 3+ color channels)

That said, I think perhaps we should rename the names of combination modes to clarify for the user what is happening.
i.e:

  • combination_mode='Add'combination_mode='sRGB_Add'
  • combination_mode='Sub'combination_mode='sRGB_Sub'

This also makes it easier to add additional combination_mode keywords later.
@JBorrow does this seem sensible to you?

¹ I would like to add an tutorial for how users may subclass MultivariateColormap to create multivariate colorbars that combine in interesting ways, and I think Oklab would make for a good example.

PS: let me know if there are any other way of color mixing that I have not thought of that you want me to try.

@JBorrow
Copy link

JBorrow commented Jul 17, 2024

Yes, this is really tricky! I agree with some very carefully chosen colour maps, you can get around this problem, and you've done a great job doing that.

The specific blending mode that I would strongly suggest is 'lighten'. This compares n items, and keeps the colour that is the lightest. This is super useful in cases where you have many colour maps (here, if I remember correctly, I was using a perceptually uniform colour map segmented into 7 pieces, each which is assigned to a specific density value which scales with its brightness in a small range), and don't want to pay the 1/n brightness price as this is a sparse situation.

Individual density maps look something like this:
test7 062349706888199

Using add:
flamingo_add

Using lighten:
flamingo_lighten

More information on this kind of visualisation is available in the yt docs (though I did not use yt for this...) https://yt-project.org/doc/visualizing/volume_rendering.html

Now, the fundamental problem with this is that it is at some level lossy. You lose information about the low-brightness items if they are 'hidden' behind high-brightness items. But I think this is an acceptable (and in many cases beneficial) choice that should be available as a core feature, and dramatically opens up the visualization possibilities.

@trygvrad
Copy link
Contributor Author

@JBorrow

Like this?
image
(source code)

We can easily add this feature, and I think we should.
However this feature can easily added in a later PR and I think it is best if we do not include it here. (This PR is large enough as it is.)
Once this PR, #28428 and the following PRs that allows for the use of multivariate colormaps in the plotting functions are accepted, I'll make a PR implementing 'lightness' and tag you.

At that point we can discuss what the name should be¹, and if we need to ship the feature with additional colormaps.

¹I would like to have combination_mode = 'choose_max' because that is descriptive of the operation we perform, but lets not make any decisions now :)

@trygvrad trygvrad force-pushed the Multivariate-colormap-types branch from 9fe5508 to b59850d Compare July 18, 2024 12:23
@JBorrow
Copy link

JBorrow commented Jul 18, 2024

Yes, that looks perfect! Thank you, definitely happy to defer this to a later PR. Just thought I would point this out as it's really a critical feature for sparse datasets.

@JBorrow
Copy link

JBorrow commented Jul 18, 2024

One thing that I will say is that there are already well-understood names for all of these modes; I would strongly suggest sticking with them. They are used in all design, drawing, and video editing programs and are well-understood.

See e.g. https://www.clipstudio.net/how-to-draw/archives/154182

Copy link
Member

@story645 story645 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to me, this def looks good enough to get in so you can build on top of and may need a refactor later. Have some minor nits, only major question is all the special casing for like byte order and the like

self._clip((X0, X1))

# Native byteorder is faster.
if not X0.dtype.isnative:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add the no-cover pragmas for now?

sub_rgba, sub_mask_bad = c._get_rgba_and_mask(xx, bytes=False)
sub_rgba = np.asarray(sub_rgba)
rgba[..., :3] += sub_rgba[..., :3] # add colors
rgba[..., 3] *= sub_rgba[..., 3] # multiply alpha
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reference / standard name for this method? It makes some sense, but is not normal compositing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also has the property that if one colormap comes back with alpha=0, then everything is 0, where I'm not sure makes sense. Should the alpha be pre-multiplied?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Truth be told, we have to treat alpha in some way, and in most use cases the implementation we choose does not matter. i.e. we can easily change this if you have an alternative expression you would like to use.

There was one use case I could think about: Two colorbars, one that is a color gradient, and another that is an alpha gradient. This implementation supports that use case, but I think it needs a bit more work to make it user-friendly.

This also has the property that if one colormap comes back with alpha=0, then everything is 0, where I'm not sure makes sense.

To me it makes sense that if alpha=0 on one colormap the result should have alpha=0. The use case I am thinking of is a categorical colomap, and you want one of the categories to hide the element (i.e. missing data).

Should the alpha be pre-multiplied?

I'm not sure what this would imply, can you describe this with an equation?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rgba[..., :3] += sub_rgba[..., :3] * sub_rgba[..., 3] is what I meant by "pre-multiply".

I am not sure what the right expression is, but would like what ever it is to be well justified. Everything I found about blending was always 2 layers (not N) so I'm not sure if there an existing generalization available to us?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any reason to do sub_rgba[..., :3] * sub_rgba[..., 3].
If we do this operation, then both [.5, .5, .5, 1] and [1, 1, 1, .5] produce the same color, and the former should always be preferred. I.e. this adds no new functionality to the multivariate colorbars.
To me, alpha has a unique use with regards to composition (i.e. layering different artists in a figure), and if we start mixing alpha and color, we interfere with this use of alpha.

The use case I have in mind is something like this:

import matplotlib.pyplot as plt
import numpy as np
import matplotlib

vals = np.zeros((256, 4))
vals[:, 3] = np.linspace(1,0, 256)
am = matplotlib.colors.ListedColormap(vals, 'alpha')
RB = plt.get_cmap('RdBu')
cmap = matplotlib.colors.MultivarColormap((RB, am), combination_mode='sRGB_add')

n = 100
x = np.linspace(-1,1, n)[np.newaxis, :]*np.ones((n,n))
y = np.linspace(-1,1, n)[:, np.newaxis]*np.ones((n,n))
r = np.sqrt(x**2+y**2)
a = np.random.random((n,n))*2-1
b = np.random.random((n,n))


fig, ax = plt.subplots()
ax.set_facecolor("k")
ax.contour(x, y, r, levels=10, colors='w', zorder=0)
cs = ax.pcolormesh(x, y, (a, b), cmap=cmap, zorder=2, vmin =(-1, 0), vmax = (1, 1))

fig.colorbar(cs.colorizer[0], ax=ax).set_label('RdBu')
fig.colorbar(cs.colorizer[1], ax=ax).set_label('Alpha')

image

Where different variables are shown using a diverging colorbar and lightness (alpha), and by using alpha, additional information can be shown behind the map (contour lines).
While this shows made-up data, I think you could make interesting maps with this. Consider for example a map of population density (alpha), median age (color) overlaid over a map of major highways.


cmaps = {
name: LinearSegmentedColormap.from_list(name, data, _LUTSIZE) for name, data in [
('2VarAddA0', _2VarAddA0_data),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are pretty esoteric names; are they documented somewhere? Might even need a comment here as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@QuLogic thanks for all the comments
These are placeholders so that we have something to test with for this PR.
We should have a longer discussion about them in a later PR, which will also introduce more of them.
see this comment for a rough timeline #28428 (comment)
(I have been thinking we need sets from 2-8 colors)
At that point we should decide on a naming scheme.

I wrote a blogpost about their design here: https://trygvrad.github.io/multivariate-colormaps-for-n-dimensions/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So one of the reasons I suggested we maybe pull these in as dark mode cmaps #28454 (comment), and I'm okay w/ that happening in a later PR, is that we could then probably have more semantically meaningful names based on the individual maps.

trygvrad and others added 9 commits August 16, 2024 11:06
Also removed all set_ functions, and replaced them with with_extremes()
This renames the 'combination_mode' keywords from 'Add' and 'Sub' to  'sRGB_add' and 'sRGB_sub'. This change is intended to provide increased clarity if/when additional keywords are added.
Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com>
@trygvrad trygvrad force-pushed the Multivariate-colormap-types branch from f4361ed to a35bc29 Compare August 16, 2024 09:26
@trygvrad
Copy link
Contributor Author

@QuLogic I rebased this, do you want to take another pass and possibly approve this?

Copy link
Member

@QuLogic QuLogic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly typos and minor items, plus you should finish the _repr_png_ test noted above.

Comment on lines 2031 to 2032
patch : nparray of shape (k, k, 3)
This patch gets supersamples to a lut of shape (N, M, 4)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See question above.

trygvrad and others added 3 commits August 19, 2024 09:08
Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com>
also allows SegmentedBivarColormap to take non-square patches.
This was easier that testing that the input is always square.
@trygvrad trygvrad force-pushed the Multivariate-colormap-types branch from b317208 to dcf6f8a Compare August 19, 2024 08:48
@trygvrad
Copy link
Contributor Author

@QuLogic thank you for looking at this again, I added a test for repr_png :)

patch : np.array of shape (k, k, 3)
This patch gets supersampled to a lut of shape (N, M, 4).
patch : np.array
Patch is required to have a shape (k, l, 3), and will get supersampled
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to be (k, k, 3) not (k, l, 3)? Sorry, still not sure what k is; is it saying it must be square?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It used to be, but I updated the code so that it also works with non-square inputs.

Copy link
Member

@story645 story645 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only requesting changes b/c I don't know the status of cm.vectormappable and don't want this going in w/ docs referring to something non-existent.

@trygvrad
Copy link
Contributor Author

@story645 all your comments should be resolved now :)
Thank you!

@story645
Copy link
Member

I can squash merge, but do you have a preference for the commit message. This is the default:

* MultivarColormap and BivarColormap

Creation and tests for classes containing multivariate and bivariate colormaps.

* __getitem__ for colors.BivarColormap

Adds support for __getitem__ on colors.BivarColormap, i.e.: BivarColormap[0] and BivarColormap[1], which returns (1D) Colormap objects along the selected axes

* Better descriptors for 'Add' and 'Sub' in colors.MultivarColormap

* minor fixes for MultivarColormap and BivarColormap

removal of ColormapBase
Addition of _repr_png_() for MultivarColormap
addition of _get_rgba_and_mask() for Colormap to clean up __call__()

* Allows one to not clip to 0...1 in colors.MultivarColormap.__call__()

Also adds a an improvement to colors.BIvarColormap.__getitem__() so that this returns a ListedColormap object instead of a Colormap object

* Multivariate and bivariate resampling of colormaps

Also removed all set_ functions, and replaced them with with_extremes()

* Corrected stubs for mutlivariate and bivariate colormaps

* minor fixes to colors.py

* Rename 'Add' to 'sRGB_add'

This renames the 'combination_mode' keywords from 'Add' and 'Sub' to  'sRGB_add' and 'sRGB_sub'. This change is intended to provide increased clarity if/when additional keywords are added.

* name as keyword variable and additional tests to multivariate colormaps

* Apply suggestions from code review

Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com>

* Fixes based on feedback on review

* removed exposure of multivariate and bivariate colormaps in cm.globals()

* improved docstring for MultivarColormap

* Apply suggestions from code review

Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com>

* test _repr_png_() for MultivarColormap

* improved docstring for SegmentedBivarColormap()

also allows SegmentedBivarColormap to take non-square patches.
This was easier that testing that the input is always square.

* updated default arguments in docs for multivar and bivar colormaps

based on code review by @story645

---------

Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com>

@trygvrad
Copy link
Contributor Author

I was thinking simply

* MultivarColormap and BivarColormap

Creation and tests for the classes MultivarColormap and BivarColormap.

But I have no strong preferences, so do what is right for the project :)

@story645 story645 merged commit b01462c into matplotlib:main Aug 23, 2024
41 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Waiting for author
Development

Successfully merging this pull request may close these issues.

Feature request: Bivariate colormapping
6 participants