The push for viridis and all of our "marketing" has been that it is color-blind friendly. So, I think we are going to need a description of how viridis is deficient and how cividis corrects for that (not in the what's new, but certainly here in the Pull Request discussion).

I haven't done a side-by-side comparison yet, but if the differences are slight enough, then maybe we could see this as a patch on viridis? There is so much momentum behind viridis that I would hate to add confusion with another very similar colormap with a very similar name.

Cividis is actually a corrected version (which is why its name is derived from viridis + cvd), done by simulating viridis as seen by someone with complete red-green colorblindness then ensuring the map had a perceptually uniform change in hue across the length of the colormap and maximizing the lightness change from the left side (lowest values) of the colormap to the right side (highest values). In the below figure, you’ll find an example of the same NanoSIMS image overlaid with a) grayscale, b) viridis, c) viridis as viewed by those with complete red-green colorblindness, and d) cividis.

The main advantage of cividis is that it is optimized specifically for viewing with a deuteranomaly/red-green colorblind. It is still useful to have both maps available but the colorblind-friendly option could be attractive to many users.

Also, cividis has 2 colors rather than 3 as seen in viridis, which can be preferred in some cases.

So, viridis is still color-blind friendly, just cividis is "more" friendly? I should note we have several other colormaps that were considered such as "plasma" that were all derived from the same methodology and colorspace.

Also, you should add an entry for this colormap in the colormap gallary page (I am having trouble finding it at the moment, though...)

First thanks a lot for your potential contribution. I think folks around here are very open to new colormaps and to making life easier for the colorblind. But more information would be helpful in assessing if this is really an improvement.

Does cividis look just like (d) to the colorblind? I'm not seeing a huge difference between (c) and (d) in terms of something that might cause me to misinterpret the data (I don't think I'm colorblind).

Good "mathematical models" of natural phenomena have errorbars in the parameters; are (c) and (d) significantly different within those error bars? As @WeatherGod says above, virdis was designed to be viewable by the colorblind using a mathematical model.

https://github.com/matplotlib/matplotlib/blob/master/examples/color/colormap_reference.py ?

Also, Whats New entries go in the directory: https://github.com/matplotlib/matplotlib/tree/master/doc/users/next_whats_new

http://matplotlib.org/devel/index.html

The point is evidently to have a colormap that looks the same to those with and without colorblindness. This seems to me like a reasonable thing to do, though the real need for it may be modest. Comments, @njsmith?

Yep! It was specifically optimized with colorblindness in mind. And yep, with this map, everyone (with normal/trichromatic color vision, deuteranomaly, and complete red-green colorblindness) sees exactly the same image.

When comparing c and d, Cividis has a steeper lightness ramp, allowing the edges of the colormap to be further apart.

Viridis is great for those with a color vision deficiency, I am not saying it isn't. I completely agree. This is simply an alternative colormap specifically designed for those with CVD.

OK, fair enough, I agree there is value in having a colormap that looks the same to more people. 👍

@jamienunez All those build tests need to be fixed. Probably just because you put the WhatsNew in the wrong spot. let us know if you need help wading through that.

@jamienunez I don't want to go all reviewer 3 on you, but I have some questions :-)

• In figure (d) above, are you showing cividis directly, or cividis that's been passed through a red-green colorblindness simulation algorithm?

• when you say "complete red-green colorblindness", are you talking about deuteranopia or protanopia?

• you say that everyone sees exactly the same thing with this colormap, but I don't understand how that's possible with anything except a greyscale colormap... are you claiming that if you take the colormap and run it through your CVD simulation algorithm repeatedly, then nothing changes? (i.e., it's a fixed point of the algorithm?) And furthermore, it's a simultaneous fixed point for all CVD variants?

• speaking of greyscale colormaps, what advantage do you see this as having over a perceptually uniform greyscale colormap?

• what algorithm exactly are you using to simulate CVD, and compute color deltas? Machado et al (2009) applied to sRGB and then converting to CAM02-UCS and computing Euclidean distance in that space?

• In your figure, the perceptual delta curve in panel (c) is not flat, but the deviation from flatness seems small (or perhaps that's an artifact of overly large y-limits?). So far as I know, no-one has ever validated a color delta algorithm on people with CVD, plus there are many variants of CVD, plus CVD simulation algorithms are fairly approximate things, plus CAM02-UCS is itself only an approximation. How confident are you that the difference in perceptual delta curves between panels (c) and (d) is actually real given all of these approximations, and what gives you this confidence?

An additional point that I just remembered about the development of viridis: we chose to clip the luminance range so that it didn't go from complete black to complete white on the ends. The reason for this was to leave room at the ends so that an image could still be annotated by black (or white) text, markers and/or lines. It seems that cividis sacrifices that for maximum perceptual delta?

From my point of view grey is usually perfectly acceptable as a non-diverging colormap when perceptual uniformity is desirable. I’ll often use jet when I don’t want perceptual uniformity (horrors!). I too wonder what the perceived advantage of the new colormap is relative to grey.

To me the real challenge is a decent looking diverging colormap that is also colourblind friendly.

Sorry for the delay in answering questions!

@njsmith:
In figure (d) above, are you showing cividis directly, or cividis that's been passed through a red-green colorblindness simulation algorithm?
Panel (d) shows cividis directly, though images below show it (again) passed through the simualtion.

when you say "complete red-green colorblindness", are you talking about deuteranopia or protanopia?
Deuteranopia is what cividis was optimized for but simulating cividis with protanopia does not make a significant difference. Below is how cividis looks to each of the (dichromat) types of CVD (severity = 100). As I mention below, I'm using colorspacious.

you say that everyone sees exactly the same thing with this colormap, but I don't understand how that's possible with anything except a greyscale colormap... are you claiming that if you take the colormap and run it through your CVD simulation algorithm repeatedly, then nothing changes? (i.e., it's a fixed point of the algorithm?) And furthermore, it's a simultaneous fixed point for all CVD variants?
Great question. You are right, it is not exactly the same and I apologize for not explaining this better. This was something I had thought a while back but checked the difference between (i) cividis and (ii) cividis fed back through the CVD simulation again and found the maps to be indistinguishable by eye but the actual difference (subtracting one from the other) was about 11. This means it isn't exactly the same (as I mistakenly stated) but they are indistinguishable. I can regenerate and post the comparison here if needed.

speaking of greyscale colormaps, what advantage do you see this as having over a perceptually uniform greyscale colormap?
Another great question. @jklymak, this answer should answer your question as well. If you compare panel a to panel b-d, are there features you can easily see in the colored maps you don't see in greyscale? Humans can only distinguish about 30 shades of grey whereas we can distinguish between millions of colors. This is the advantage of cividis over greyscale: the colors are more easily distinguishable from one another, allowing for better sensitivity between the shades of each pixel colored with cividis.

what algorithm exactly are you using to simulate CVD, and compute color deltas? Machado et al (2009) applied to sRGB and then converting to CAM02-UCS and computing Euclidean distance in that space?
We use colorspacious! All of this started with seeing your presentation about viridis/colorspacious/viscm so we use the same approach to calculate perceptual delta (so yes to the method you mention in your question). It made our team wonder if a map like viridis can be automatically made using pre-existing maps while also optimizing for CVD (Hence the creation of cividis, viridis + cvd).

In your figure, the perceptual delta curve in panel (c) is not flat, but the deviation from flatness seems small (or perhaps that's an artifact of overly large y-limits?). So far as I know, no-one has ever validated a color delta algorithm on people with CVD, plus there are many variants of CVD, plus CVD simulation algorithms are fairly approximate things, plus CAM02-UCS is itself only an approximation. How confident are you that the difference in perceptual delta curves between panels (c) and (d) is actually real given all of these approximations, and what gives you this confidence?
Let me get back to you on this later today. I have an example that shows the difference but, looking at it now, am unsure if it's just me that sees the difference out of bias or if it really is a quality example. I need to wait for a team member to come in and double check they see the same thing.

@WeatherGod:
An additional point that I just remembered about the development of viridis: we chose to clip the luminance range so that it didn't go from complete black to complete white on the ends. The reason for this was to leave room at the ends so that an image could still be annotated by black (or white) text, markers and/or lines. It seems that cividis sacrifices that for maximum perceptual delta?
Cividis does not actually fade to black or white at its ends. Please refer to the additional images I supplied above in this post. For a NanoSIMS image, I also have images where I overlaid cividis on cell biomass then made the rest of the image black. The black background is easy to see even when compared to cividis's lowest value (the dark blue). I am unsure if I can share this, though, since the image is for a paper. I'll ask and post it (or email it to you) if you're interested in seeing it. Or I can just generate an example with published data.

@jklymak:
To me the real challenge is a decent looking diverging colormap that is also colourblind friendly.
We are actually looking into adding this as a utility in the module being released with our paper! Since we automated the full colormap-optimization technique, it'll be an easy component to add. The code is open source and can be found here. Please don't judge the code too harshly (though input is very much appreciated) since it hasn't been beautified for the public quite yet. This is something I need to do before the paper is published.

If anyone is interested in seeing the paper, please let me know and I'll see what I can do. The paper, of course, can not be shared outside this group since it is still in review.

Oh and @jklymak, I am changing colormap_reference.py and the correct what's new now.

@njsmith, to answer your last question (now that a couple colleagues have verified my observations):
Good Colour Maps: How to Design Them by Peter Kovesi in 2015 discusses colormap design principles and the differences between colormaps. If you look at the beginning of section 2, a test image is discussed that allows a direct visual for how a colormap changes along its length. When comparing the complete red-green colorblind version of viridis (top colormap in the image below) to cividis (bottom colormap), we got the following image. Though viridis looks great, cividis (especially in the center) has better definition.

This can be used as a direct evaluation for comparing the small change in perceptual deltas. Higher delta: better definition. Also, there are some places where the sine waves in the top image don't line up exactly, showing slight changes in perceptual delta.

I feel this is a great way of supporting the small perceptual delta difference and the overall method of conversion between spaces. Please let me know what you think!

The doc-building doesn't like the unicode in your name. Umm, not 100% sure whats to be done about that. Someone smart will:

Warning, treated as error:
/home/circleci/project/doc/users/next_whats_new/new_color_cycle.rst:11:undecodable source 
characters, replacing with "?": b'\tNu>>>\xf1<<<ez J, Anderton C, and Renslow R. Optimizing 
colormaps with consideration for color vision deficiency to enable accurate interpretation of scientific 
data. 2018.'

That's okay. Both spellings are fine so I removed it. Hopefully that allows the tests to pass.

Can someone look into the cause of these tests failing? I can't find the source of the error

That is a great point @WeatherGod. Viridis is 'colorblind friendly’ so your advertising was accurate. However, being ‘colorblind friendly’ does not mean it was fully optimized for CVD. So while it performs much better than nonfriendly colomaps, like rainbow, it still has some drawbacks in the CVD space. People with CVD could perceive the underlying data differently than people without CVD. The goal with cividis was to ensure as-identical-as-possible perception of the data, while maintaining perceptual uniformity and linear lightness. Viridis is still useful since it is more aesthetically pleasing, is still perceptually uniform for noncolorblind individuals, and cycles through three colors which amplifies differences between pixels close in value.

Below, I have attached another image that shows the perceived vs true difference between neighboring pixels (values along the white line are plotted). Our paper describes this plot in further detail so we have submitted it to arXiv and I will share the link once it is available. Essentially, these plots show that for people with CVD, some small changes in the underlying data may be represented by a relatively larger change along the colormap (i.e., disproportionate changes), and vice versa. Greyscale and cividis maintain proportional changes throughout (as does viridis, but only for people without CVD).

Based on this, what would be the best way to word this in the “what’s new” file? I could change it to the following: “cividis is a new perceptually uniform colormap with linear lightness, which maintains proportionate perception for both people with and without colorblindness (CVD).“

travis CI had some sort of problem....

I'm still not quite following why viridis doesn't have the same perceptual deltas for colorblind viewers. I still suspect its a difference in the models used to simulate colorblindness. W/o a parameter study, i.e. where you make a cloud of such plots by Monte Carlo changing the model parameters over some reasonable range, I am not clear if the differences in your plot above are significant or not.

I don't mind this going in because it looks the same for colorblind and non-colorblind viewers (within any model uncertaintties), and that seems useful. But it would be educational to know what the difference in models is, particularly before you claim yours "corrects" viridis.

@jklymak:
The model used to simulate colorblindness for both viridis and cividis should be the same. I used the module @njsmith's group made which does this part automatically. (I think) Viridis was not designed to have the same perceptual delta for normal color vision and CVD which is what causes the different deltas. Different colors perceived can lead to different perceptual deltas.

Edit: I understand now what you mean by the need for a parameter study! Sorry about the misunderstanding. This is something my team should indeed look into so we will try to do so soon. I'll let you all know what we find.

I should say, I think it's super cool that you're working on this, and I hope my comments aren't discouraging – if one of the outcomes of the viridis work is that people are thinking about colormaps more thoughtfully and critically, then that's awesome.

My main concern is that since there's so much confusion out there about colors, matplotlib should be very careful about exactly what we communicate and what recommendations we make. In particular, a statement like "colormap X has advantages for CVD users" can easily lose all nuance and then turn into a bludgeon people use on each other, so we should be careful not to make such statements unless we can clearly explain why we think that and what the trade-offs are.

You are right, it is not exactly the same and I apologize for not explaining this better. This was something I had thought a while back but checked the difference between (i) cividis and (ii) cividis fed back through the CVD simulation again and found the maps to be indistinguishable by eye but the actual difference (subtracting one from the other) was about 11.

Yeah, the reason this jumped out at me is that I also managed to confuse myself about this at one point :-).

I do think you should be making a habit of including CVD-cividis in any comparison images/graphs. Comparing CVD-viridis to non-CVD cividis is conceptually the wrong thing, and this would let you demonstrate the similarity between non-CVD cividis and CVD-cividis instead of just asserting it.

Humans can only distinguish about 30 shades of grey whereas we can distinguish between millions of colors. This is the advantage of cividis over greyscale: the colors are more easily distinguishable from one another, allowing for better sensitivity between the shades of each pixel colored with cividis.

I see what you're trying to say, but this summary is pretty misleading. Humans can distinguish millions of colors, sure, but we're not putting millions of colors in any single colormap. (Not even jet does that :-).) A colormap is a 1-dimensional line drawn through a 3-d volume; it's not the size of the volume that matters, it's the length of the line.

When you use viscm to examine a colormap, one of the things it calculates is its arc-length (in CAM02-UCS). The idea is that this hopefully gives some clue about how much sensitivity the colormap has – longer colormaps ought to let users make finer discriminations. (This number out to be somewhat misleading for reasons I'll discuss below, but it's the best quantitative measure we have for this.) When I put cividis into viscm, It reports an arc-length of 99.5. For comparison, a perceptually uniform ramp from black→white has length 100.0 on the same scale (and viridis has length 123.9), so CAM02-UCS is claiming that cividis and greyscale actually have almost identical sensitivities.

And unfortunately, as I said above, there's actually a good reason not to trust CAM02-UCS here. As Kovesi points out in his colormaps paper, the human perception experiments used to calibrate CAM02-UCS and similar scales use large flat patches of color (low spatial frequency), while colormaps are generally used to display data with high-spatial frequency. At high spatial frequencies the visual system shifts its emphasis more towards the lightness channel and away from hue/saturation. So using CAM02-UCS to estimate colormap sensitivity will tend to be biased, in that it will underestimate the size of lightness changes, and overestimate the size of hue/saturation changes. Since cividis has more hue/saturation variation than greyscale, this means that our estimate above is overestimating the sensitivity of cividis and/or underestimating the sensitivity of black→white.

(This also matches our experience designing colormaps: we made various attempts to maximize arclength by using huge movements in the a'b' plane, and this gives colormaps that sound great on paper, but when we tried using them for real images they really weren't any better at resolving fine features than greyscale was. We found this very confusing until we read Kovesi's paper! It's kind of depressing: we have all this awesome color theory and color technology, yet at the end of the day, apparently the main benefit of using non-grey colors is that they're pretty and people like them, and mostly what all the theory does is help us design pretty maps that aren't worse than greyscale. But so it goes...)

Anyway, what I take from this is that compared to cividis, a perceptually uniform greyscale map using the whole black→white range actually (a) gives better sensitivity, (b) handles black/white printing better (by definition), (c) does an even better job of looking the same to users with CVD.

This doesn't necessarily mean cividis is useless or shouldn't be included in matplotlib – it seems like a reasonable black→yellowish map, and it's great to have more options for colorblind users. But if we want the docs to say "We think cividis is the best choice in the following situations: ____" then I'm having trouble thinking of ways to fill in the blank.

When comparing the complete red-green colorblind version of viridis (top colormap in the image below) to cividis (bottom colormap), we got the following image. Though viridis looks great, cividis (especially in the center) has better definition.

I'm really struggling to see consistent differences between these two images. This is way ambitious for this PR :-), but it'd be really interesting someday to do a double-blind study to see how well these colormaps actually work for real people and real equipment, instead of just trusting the computational models. Maybe one could show people Kovesi's test image and ask them to annotate how far down each line seems to go on their monitor, then compare the places where they click between different colormaps and for folks with or without CVD.

b'\tNu>>>\xf1<<<ez J

I think that's saying that the ñ in the file was encoded using Latin-1. Sphinx is probably expecting UTF-8 – most things do, these days. Possibly your editor needs to have some setting tweaked to use UTF-8?

@njsmith: thank you very much for your feedback and comments! You make really great points, such as how cividis may actually compare grayscale. My team and I will look into addressing these, along with the parameter sweep, soon.

Is there anything else I need to do at this point for cividis in matplotlib?

@jamienunez Is it OK to push to your branch directly? You've got rand.random and it should be random.rand. Maybe you copied from whatever I quickly wrote down, but...

Okay I made the change and added plt as the name for matplotlib.pyplot

And yep, no worries if you want to make edits yourself! I'm fine with changes being made either way.

If we do merge cividis, then I suggest phrasing like:

"A new black/yellow colormap named 'cividis' was added. Like viridis, cividis is perceptually uniform and colorblind friendly. However, cividis also goes a step further: not only is it usable by colorblind users, it should actually look effectively identical to colorblind and non-colorblind users. For more details, see Nuñez J, Anderton C, and Renslow R. (submitted). Optimizing colormaps with consideration for color vision deficiency to enable accurate interpretation of scientific data."

@jamienunez You know, I was thinking about this more today, and it occurred to me that it might not actually be too hard to directly design colormaps with the constraint that they remain invariant under (simulated) CVD. The matrices that implement the Machado et al transform have a well-defined invariant 2d subspace:

In [4]: np.linalg.eig(colorspacious.machado_et_al_2009_matrix("deuteranomaly", 100))
Out[4]: 
(array([  5.73655225e-07,   1.00869758e+00,   1.00000585e+00]),
 array([[ 0.92205474, -0.64899895,  0.5774051 ],
        [-0.38601936, -0.60561839,  0.57737375],
        [ 0.02835681, -0.4604636 ,  0.57727195]]))

notice that the 2nd and 3rd eigenvalues are essentially 1, meaning that the plane spanned by their corresponding eigenvectors is invariant under this transformation. The 3rd eigenvector is all-constant; that's the greyscales. The 2nd eigenvector [-0.65, -0.61, -0.46] is more interesting – it gives us the Machado et al estimate of where to find the blue/yellow line that deuteranopes can perceive normally. So for example, this means that the dark blue at [-0.65, -0.61, -0.46] + 0.65 = [0, 0.04, 0.19] is invariant, and so are any rescalings of it. (That times 2 is quite a nice royal-blue-ish color.) -1 * [-0.65, -0.61, -0.46] + 0.35 is a kind of pleasant pale khaki. And linear interpolation between any of these points gives us a color gradient that falls entirely within the Machado et al invariant subspace. Of course it's not perceptually uniform, but that's easily fixed by converting into CAM02-UCS and then parametrizing by the arclength...

What about other forms of CVD? In the Machado et al model, all degrees of deuteranomaly have the same invariant subspace as full deuteranopia:

In [5]: np.linalg.eig(colorspacious.machado_et_al_2009_matrix("deuteranomaly", 50))
Out[5]: 
(array([ 0.30808704,  1.00428102,  1.00000394]),
 array([[ 0.93252489, -0.64852155,  0.5774164 ],
        [-0.35994545, -0.60532024,  0.57737901],
        [ 0.02892397, -0.46152703,  0.57725539]]))

Protanopes have a slightly different 2nd eigenvector, but it's close:

In [6]: np.linalg.eig(colorspacious.machado_et_al_2009_matrix("protanomaly", 100))
Out[6]: 
(array([ -1.44236937e-07,   9.90565816e-01,   9.99999328e-01]),
 array([[ 0.98961055, -0.62410276,  0.57735461],
        [-0.14374425, -0.59549249,  0.57735172],
        [-0.00292275, -0.50585022,  0.57734448]]))

So one could just compromise between them, or just use the deuteranopia one since it's more common.

I'm not sure how helpful this is – for all I know you're at the point where you've written the paper and just want to move on, and it's anti-helpful. (Sorry!) But if you want to talk more about this then we could switch to email or something instead of cluttering up this PR.

@WeatherGod: We aren't planning to change cividis but, just in case, we can wait to add it to matplotlib when our paper has been published. It's very unlikely to change after that point. We anticipate reviewer comments on the paper before the end of the year.
@njsmith: We actually had a very similar idea we've started on and were hoping to discuss it with you. I'll email you so we can discuss this further!

I am making an executive decision and merging this for the 2.2RC1.

@jamienunez If you want to tweak the colors, we can do that before 2.2 final release (likely in a week or two).

Thank you!