How is this different that the over/under functionality we already have?

Do you mean Colormap.set_over() ? It is different in how BoundaryNorm() is choosing the colors within the levels' range. If you set_over(), the color of the last box before the extension will still be the last color of the colormap.

It is probable that I missed the way to do what I need without this PR, but I couldn't find any example in the gallery. @efiring seemed to agree on the rationale of this PR, but if proven useless I have no problem to close it (I just found it quite useful myself).

Sorry, left a comment without reading enough context 😞

The point of this is to reserve levels in the normalization to use as the over/under colors ? I see how to do this using existing tools and it is annoying.

How does this interact with the clip kwarg?

That's a good point: using clip=True and anything else than extend='neither' makes no sense, unless you want to clip the data and keep the normalized colors of the example 2 above, which is a very strange use case.

Currently something silently happens if you set extend and clip together, but it seems wrong. Should I force extend to "neither" if clip==True, or should I take care of the use-case above?

(I hope its clear enough, otherwise I can provide an example)

I am generally in favor of noisy exceptions (ex if extend is not 'neither', raise if clip is True).

Instead of calling this 'extend' in the normalization class, how about 'open_ends' or something like that? It seems what this is really specifying is if there should be an implicit +/- inf on the edges of the boundary list.

I agree with raising an exception.

The reason for calling it extend= is that it really merges the concepts of BoundaryNorm and Colorbar. If you recall the gallery example http://matplotlib.org/examples/api/colorbar_only.html (example 3), the keywords "boundaries" and "ticks" are actually redundant (because ColorbarBase knows about the norm properties).

Ideally, in my example above, the extend keyword in the call to ColorbarBase should be unnecessary. I think that it is possible to do this automatically but I didn't want to invest too much time in it without asking you first.

That is exactly why I don't want to call it extend. My knee-jerk reaction to reduce the coupling between the normalization classes and the colorbar base class.

I am in general very very wary of doing things automatically, but can be convinced otherwise.

:D

From my point-of-view (quite new to python after a long time with IDL), the example of http://matplotlib.org/examples/api/colorbar_only.html is horribly wordy. In fact, I assume that it was written in times before that the ColorbarBase base class was able to get all these infos from the normalization classes. It seems that the coupling you are complaining about already occurred ;).

For my own purposes I made a small library which adds this functionality and adds some higher level wrappers in order to prevent any mismatch between data, plot, and colorbar, which seems to be quite easy to occur with matplotlib.

But one argument in favour of decoupling however is that various functions seem to react differently to the norm class.

I'll misuse another example (http://matplotlib.org/examples/images_contours_and_fields/pcolormesh_levels.html). I just changed the levels in order to make use of extend='both':

import matplotlib.pyplot as plt
from matplotlib.colors import BoundaryNorm
import numpy as np


# Make the data
dx, dy = 0.05, 0.05
y, x = np.mgrid[slice(1, 5 + dy, dy),
                slice(1, 5 + dx, dx)]
z = np.sin(x) ** 10 + np.cos(10 + y * x) * np.cos(x)
z = z[:-1, :-1]

# Z roughly varies between -1 and +1
# my levels are chosen so that the color bar should be extended
levels = [-0.8, -0.5, -0.2, 0.2, 0.5, 0.8]
cmap = plt.get_cmap('PiYG')
norm = BoundaryNorm(levels, ncolors=cmap.N, extend='both')

# Plot 1
plt.subplot(2, 1, 1)
im = plt.pcolormesh(x, y, z, cmap=cmap, norm=norm)
plt.colorbar(extend='both')
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.title('pcolormesh with levels')

# Plot 2
plt.subplot(2, 1, 2)
plt.contourf(x[:-1, :-1] + dx / 2.,
             y[:-1, :-1] + dy / 2., z, levels=levels,
             cmap=cmap, extend='both')
plt.colorbar()
plt.title('contourf with levels')

plt.show()

The two plots are equivalent and correct but for contourf, the extend call occurs in contourf, while for pcolormesh it has to occur at the colorbar call.

If I make no call to extend at all but choose clip=True (previous):

norm = BoundaryNorm(levels, ncolors=cmap.N, clip=True)

# Plot 1
plt.subplot(2, 1, 1)
im = plt.pcolormesh(x, y, z, cmap=cmap, norm=norm)
plt.colorbar()
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.title('pcolormesh with levels')

# Plot 2
plt.subplot(2, 1, 2)
plt.contourf(x[:-1, :-1] + dx / 2.,
             y[:-1, :-1] + dy / 2., z, levels=levels,
             cmap=cmap)
plt.colorbar()
plt.title('contourf with levels')

It is amusing to see that contourf chooses different colors (somehow in accordance with my use-case for extend?) and marks out of bounds with a white color.

To conclude: I'd prefer to have more control of the Normalize class on the plotting functions because the two are inherently related, BUT I can understand that this is going to be a mess if we start to change things...

Sorry for my confusing post above. I'll get back to you tomorrow after thinking about all this a bit more ;-)

In your second example, pcolor is doing what I expect for clip=True . In the contourf case the white areas at the high/low ends I suspect are np.nan which is mapped to 'bad' which iirc maps to transparent. The color that contourf hands to the normalization class is the center of the bands, not the edges, hence why it picks the same colors for the valid data in both cases.

In this case where you are using boundary norms the tight coupling is natural, but say you want to use contourf with a non-linear color map, but with linear levels.

Yes sorry, the contourf example was OT because it has nothing to do with the normalization.

I see two solutions:

1. If we call the keyword "extend", it has to be taken into account by all functions that take norm=* as argument. Otherwise it is misleading, since all other properties (such as bounds, colors and clipping) are already derived from the normalization class as shown by the pcolormesh example above. I largely prefer this solution because it is consistent and less buggy. However, I have too little experience with mpl to estimate the amount of changes that it would represent.
2. We call it something else and leave it as is it now. It would remain a minimal change and would maybe require a new gallery example to illustrate its use.

What do you think?

I don't quite understand 1. In both cases the plotting function takes in some data, does some computation, and then gets a scalar out. This scaler is then passed to the norm to be re-scaled to [0, 1] (or [0, N]) and the rescaled value is then passed to cmap to be turned into an RGBA value. I don't think there is any case where the plotting functions are introspecting anything about norm. In the pcolormesh case each square is just going through that mapping process.

Yes, ok. I don't understant all the details of what is going under the hood when a call to pyplot.colorbar is made:

norm = BoundaryNorm(levels, ncolors=cmap.N, extend='both')

# Plot 1
plt.subplot(2, 1, 1)
im = plt.pcolormesh(x, y, z, cmap=cmap, norm=norm)
plt.colorbar()
plt.title('pcolormesh, forgot extend to colorbar')

# Plot 2
plt.subplot(2, 1, 2)
im = plt.pcolormesh(x, y, z, cmap=cmap, norm=norm)
plt.colorbar(extend='both')
plt.title('pcolormesh with levels')

So the plot is fine with my new implementation, so it's really just about the colorbar. I guess that since the colorbar knows the levels I've chosen, it must know about the normalize object I gave to pcolormesh.

You will have to ask @efiring about the colorbar code.

Ok, the change needed to make it do what I needed was minimal, but I have too little overview of the whole thing to judge if it's a good thing or not. If find it more consistent since no double call is needed, but I can understand that it is quite a specific requirement of mine.

By adding:

if hasattr(norm, 'extend') and norm.extend != 'neither':
    extend = norm.extend

in ColorbarBase's init I am forcing the keyword to what I want. If you agree on the "intrusion" I will add a few tests, if not we should decide on a name for the keyword.

My previous examples now look like this:

""" Illustrate the use of BoundaryNorm wht the "extend" keyword """

import matplotlib.pyplot as plt
import matplotlib as mpl

# Make a figure and axes with dimensions as desired.
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(6, 2))

# Set the colormap and bounds
bounds = [-1, 2, 5, 7, 12, 15]
cmap = mpl.cm.get_cmap('viridis')

# Default behavior
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap, norm=norm, extend='both',
                                orientation='horizontal')
cb1.set_label('Default BoundaryNorm ouput');

# New behavior
norm = mpl.colors.BoundaryNorm(bounds, cmap.N, extend='both')
cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap, norm=norm,
                                orientation='horizontal')
cb2.set_label("With new extend='both' keyword");
plt.tight_layout()
plt.show()

import matplotlib.pyplot as plt
from matplotlib.colors import BoundaryNorm
import numpy as np

# Make the data
dx, dy = 0.05, 0.05
y, x = np.mgrid[slice(1, 5 + dy, dy),
                slice(1, 5 + dx, dx)]
z = np.sin(x) ** 10 + np.cos(10 + y * x) * np.cos(x)
z = z[:-1, :-1]

# Z roughly varies between -1 and +1
# my levels are chosen so that the color bar should be extended
levels = [-0.8, -0.5, -0.2, 0.2, 0.5, 0.8]
cmap = plt.get_cmap('PiYG')
norm = BoundaryNorm(levels, ncolors=cmap.N, extend='both')

# Plot 1
plt.subplot(2, 1, 1)
im = plt.pcolormesh(x, y, z, cmap=cmap, norm=norm)
plt.colorbar()
plt.title('setting extend=both is obsolete')

# Plot 2
plt.subplot(2, 1, 2)
im = plt.pcolormesh(x, y, z, cmap=cmap, norm=norm)
plt.colorbar(extend='neither')
plt.title('extend=neither however is ignored')

plt.tight_layout()
plt.show()

Sorry to have let this slide for so long; I will try to spend some time on it today. Something along these lines will be good, but I think there might be a better approach: a norm kwarg (or pair of kwargs) that changes the target range from 0-1 to some other range. I think this might handle the use case for this PR in a very general way (e.g., by setting the target range to [0.1, 0.9]), and as a bonus, handle some other use cases at the same time.

My drive-by comment on this is that there should be a way to do this without breaking any existing code. If the user says extend='neither' then they should get neither. I think this can be done by changing the default to None, if it is None then consult the norm then fall back to the current default. If the user passes in a non-None value, respect it.

What I'm suggesting does not inherently involve an "extend" kwarg, or any magic; it would be entirely explicit, and would have no effect on existing code. It would be independent of the "extend" kwargs in colorbar and contourf.

@efiring Any update on this?

I'll make a point of getting back to this and other color questions no later than this weekend. Thanks for the reminder.

Sorry I was not able to be more helpful on this one. Let me know if I can do anything.

As a side note, xray already implements the logic I'm asking for:

import xray
a = xray.DataArray([[1,2,3],[4,5,6]])
a.plot(levels=[2,3,4,5]);

Will produce (note the colorbar colors):

Maybe @shoyer can comment on this (no need to read all my lengthy examples, the original post will suffice).

This looks consistent with the way contourf works. I like it. @fmaussion, would you make the one change suggested by @tacaswell, please? If I understand correctly, the point is that if the call to colorbar includes an extend kwarg, then that should be respected, regardless of whether the norm also has an extend kwarg. It's only a subtle difference and small change from what you have.

Yes, I think I agree with this change. With xray, we did some work to make an external wrappers for pcolormesh, imshow and contourf that handles levels, extend and plot bounds that work identically for each plot type (although the underlying plot is produced differently). All this code exists in our plotting module and you're free to adapt anything you like, of course (we use an Apache license).

@efiring OK, I'll get back to this soon.

I've incorporated @tacaswell requirements and added an image test (see the image here)

The failing test seems unrelated to my changes: it concerns only one single test environment?

I'd like to update the examples for the documentation too but I will do this in a separate PR.

I restarted that test environment. Some of the tests can be a bit finicky on cloud environments.

@fmaussion, I have a proposed simplification of the calculation. May I push to your repo? If so, I think there is something you need to click to add me as a collaborator.

@efiring @jklymak I updated the PR with the following example which motivated this PR:

import matplotlib.pyplot as plt
from matplotlib.colors import BoundaryNorm
import numpy as np

# Make the data
dx, dy = 0.05, 0.05
y, x = np.mgrid[slice(1, 5 + dy, dy),
                slice(1, 5 + dx, dx)]
z = np.sin(x) ** 10 + np.cos(10 + y * x) * np.cos(x)
z = z[:-1, :-1]

# Z roughly varies between -1 and +1
# my levels are chosen so that the color bar should be extended
levels = [-0.8, -0.5, -0.2, 0.2, 0.5, 0.8]
cmap = plt.get_cmap('PiYG')

# Before this change
plt.subplot(2, 1, 1)
norm = BoundaryNorm(levels, ncolors=cmap.N)
im = plt.pcolormesh(x, y, z, cmap=cmap, norm=norm)
plt.colorbar(extend='both')
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.title('pcolormesh with extended colorbar')

# With the new keyword
norm = BoundaryNorm(levels, ncolors=cmap.N, extend='both')
plt.subplot(2, 1, 2)
im = plt.pcolormesh(x, y, z, cmap=cmap, norm=norm)
plt.colorbar()  # note that the colorbar is updated accordingly
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.title('pcolormesh with extended BoundaryNorm')

plt.show()

I've also authorized edits from maintainers. Please feel free to make any change.

Dear all, I am away from my laptop for the next three weeks - feel free to edit / merge as you wish.

@efiring @jklymak I'm now back at work - anything more that needs to be done here?

While trying to add the concept to this example, I noticed that the coupling between norm and colorbar in the case of contourf does not work as expected :'-( :

import matplotlib
import matplotlib.pyplot as plt
from matplotlib.colors import BoundaryNorm
from matplotlib.ticker import MaxNLocator
import numpy as np


# make these smaller to increase the resolution
dx, dy = 0.05, 0.05

# generate 2 2d grids for the x & y bounds
y, x = np.mgrid[slice(1, 5 + dy, dy),
                slice(1, 5 + dx, dx)]

z = np.sin(x)**10 + np.cos(10 + y*x) * np.cos(x)

# x and y are bounds, so z should be the value *inside* those bounds.
# Therefore, remove the last value from the z array.
z = z[:-1, :-1]

# Z roughly varies between -1 and +1
# my levels are chosen so that the color bar should be extended
levels = [-0.8, -0.5, -0.2, 0.2, 0.5, 0.8]

# pick the desired colormap, sensible levels, and define a normalization
# instance which takes data values and translates those into levels.
cmap = plt.get_cmap('PiYG')
norm = BoundaryNorm(levels, ncolors=cmap.N, extend='both')

fig, (ax0, ax1) = plt.subplots(nrows=2)

im = ax0.pcolormesh(x, y, z, cmap=cmap, norm=norm)
fig.colorbar(im, ax=ax0)
ax0.set_title('pcolormesh with levels')


# contours are *point* based plots, so convert our bound into point
# centers
cf = ax1.contourf(x[:-1, :-1] + dx/2.,
                  y[:-1, :-1] + dy/2., z,
                  cmap=cmap, norm=norm)
fig.colorbar(cf, ax=ax1)
ax1.set_title('contourf with levels')

# adjust spacing between subplots so `ax1` title and `ax0` tick labels
# don't overlap
fig.tight_layout()

plt.show()

Why is that so? Why should the signature of contourf and pcolormesh be different?

Using BoundaryNorm with contourf makes no sense at all. Maybe it should trigger a warning. I suppose the alternative you are expecting is that contourf, when given a BoundaryNorm, would take its levels from that. This seems to me like needless complexity, though.

Contourf is fundamentally based on discrete levels; pcolormesh starts out with the opposite point of view, that it is representing a continuum. BoundaryNorm provides a discretization mechanism for pcolormesh, but the discretization is built in to contourf.

Thanks for the quick reply. I will write a new dedicated example then

@jklymak and @fmaussion As far as you can see, is anything needed beyond a rebase? Are there outstanding questions still to be resolved? It would be nice to get this finished and merged ASAP so it doesn't sit around for a few more years.

Away from a real computer for a few days so don’t wait for me!

@fmaussion I would like to get this merged. Are you available to resolve the conflict and address anything else that is pending?

The present conflict is that this PR is deleting tutorials/colors/colorbar_only.py. We definitely don't want to do that.
Apart from that, what's missing is the entry in "examples" or "tutorials" illustrating the new feature here.

@fmaussion This looks to be done, but needs a rebase and the colorbar_only tutorial re-instated! Did you want to tackle it, or should one of the devs?

It also looks like the example from api_changes needs to be moved to the reinstated colorbar_only.py tutorial (or somewhere).

Closing, since I have rebased and moved this to #17534.