I think this is too complicated and confusing.
Tentative suggestions:

• Require alpha to be either a scalar or an array-like.
• If it is array-like, its shape must match the collection's colors after broadcasting.
• Apply each alpha to facecolor and edgecolor, for collections with both.
• If there is a genuine need for separate alpha scalars or arrays for facecolor versus edgecolor, then give each its own kwarg; but I would put a high threshold of acceptance on this added complexity, and would prefer that it not be included.

Even with these suggestions, I am hesitant about this. I suspect there are other multi-alpha use cases, and it would be best to identify all of them and handle them consistently.

Functionality for modifying alpha in an existing RGBA array probably will need to be factored out so it can be reused.

There is defiantly complexity here.

If I was to make the alpha input more strict by requiring its shape to match the shape of the colors then there would be a less complicated way of doing this. This leaves me unsure of how to handle the case of the user setting edgecolors and facecolors with different sized arrays. Which brings me to kwargs, and I don't think it is a good idea either.

A lot of complexity also comes from handling off cases where the colors are set to things like 'none' or 'face'.

Could you clarify what you mean in your last point about factoring out functionality? Do you mean modifying alpha in [a single] existing RGBA array or many existing RGBA arrays. If the former then that should be accomplished just calling colorConverter.to_rgba_array(). If the latter then perhaps introducing this functionality to colorConverter would be wise?

Also do you think I should consider abandoning this all together?

Factoring: I was thinking of color mapping, which operates on an array but takes only a single alpha value for the whole array. Even in the original issue, #3643, it seems like this is where the real demand is. As noted in comments there, any case in which an explicit list of facecolors or edgecolors is supplied can be handled by the user when generating the list--just supply a list of RGBA. So the one area where functionality could be added is in anything inheriting from ScalarMappable, in cases where the colors are generated by normalizing and mapping an array, c. Here, it might be reasonable to pass through an array of alpha values that matches c, so that ScalarMappable.to_rgba could apply them. The changes needed in to_rgba would be quite small, I think. The ScalarMappable subclasses would still need to have logic to ensure that non-scalar alpha is used only when mapping is being used, and that the shapes match.
There is still the question of what to do when an alpha array is specified along with color mapping for facecolors: then in the absence of an additional kwarg, the only way to specify alpha for the edgecolor is by providing an RGBA value or array of values. Maybe that's good enough. When alpha is a scalar the behavior would be unchanged, so maybe everyone would be happy.

Taking your suggestions into account, this is what I implemented. Much less complex.

For example in examples/shapes_and_collections/artist_reference.py, I can initialize the collection in the following way,

colors = np.linspace(0, 1, len(patches))
alpha = colors
collection = PatchCollection(patches, cmap=plt.cm.hsv,alpha=alpha)
collection.set_array(np.array(colors))

and be given the following result,

The ScalarMappable subclasses would still need to have logic to ensure that non-scalar alpha is used only when mapping is being used, and that the shapes match.

With regards to this. Collection was the only class I found that needed additions (likely there are more?) and in this case a mapping is being used even when no cmap argument is given in __init__
See https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/collections.py#L119, which sets cmap here, https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/cm.py#L198
So I found I couldn't check if a mapping is being used or that the shapes match but ultimately ScalarMappable.to_rgba will get called and check this.

Can you squash all of the discarded revisions out?

Getting this to work for images seems quite difficult. The difficulty is from when alpha is being applied in _make_image, the image gets resampled and that changes its shape meaning I can't apply the alpha values unless I resample the alpha values also.

This is what I had implemented before I realized the color values were given a different size. From the same function L402

if A.ndim == 2 or created_rgba_mask:
    alpha = self.get_alpha()
    if alpha is not None:
        alpha_channel = output[:, :, 3]
        np_alpha = np.asarray(alpha)
        if np_alpha.ndim != 2 and alpha != 1.0:
            # Single alpha applied to whole image
            alpha_channel[:] = np.asarray(
                np.asarray(alpha_channel, np.float32) * alpha,
                np.uint8)
        elif np_alpha.ndim == 2 and np_alpha.shape == alpha_channel.shape:
            # Array of alphas applied to each color
            alpha_channel[:] = np.asarray(
                np.multiply(np.asarray(alpha_channel, np.float32), np_alpha),
                np.uint8)

I don't think that per-pixel-alpha makes sense for images in general. As you have found, it is too complicated--and what's the use case? It might make sense for the case that can be handled by pcolormesh--that is, when there is no resampling or interpolation.

It seems really odd to me to embed a check like this is cmap that can only be used in some of it's clients.

Either alpha should become a first class property that gets folded into the Collection product logic or the alpha should go into the actual colormaps (so they do R -> R^4 instead of R -> R^3).

@tacaswell, I don't understand your either/or above. With the change in this PR, alpha is going into the colormapping--it is applying an array of alpha, if that is supplied, instead of just a scalar value. It isn't presently doing the right kind of shape checking, but it is on the right track. When that is fixed, it should work for all collections, I believe. My point is that I don't think we should try to make it work now for images, which are not collections. They are a different sort of beast, and I think that allowing alpha as an array will require changes in the extension code. Maybe that would be easy for @mdboom, maybe not.

@tacaswell, Now I think you are suggesting that the application of a varying alpha should be pushed down into the Colormap.__call__ method, correct? One way of doing this would be to make that method do exactly what is being done in the present PR whenever it gets an alpha array. It is a little bit more complicated than what is done here, but not too bad. The other would be inserting variable alpha into the _lut. We have the ability to specify a colormap that includes such variable alpha, but I don't see any simple API for injecting variable alpha into an existing colormap. In any case, this is not the type of use case addressed by the present PR, so the question is whether it better to leave the post-mapping application of variable alpha where it is in this PR, in ScalarMappable, or to push it down one level into Colormap.__call__.

Maybe the best reason for pushing it down is that this would make it possible to keep the alpha array specification from clobbering the handling of "bad" values.

Getting all of this right is looking a bit tricky, as usual with alpha handling.

This direction makes sense to me. Thank you for the feedback 👍