Dominant colour of an image with an alpha band. #3692

kipcole9 · 2023-09-27T22:57:06Z

kipcole9
Sep 27, 2023

I am calculating the dominant color based upon this example. No issues when the image does not have an alpha band. But I'm not conceptually sure how to manage an image with an alpha band.

Flattening the image will understandably mean that the background colour will dominate the transparent areas. And in many images that are designed to be composited that means the dominant colour often becomes that value (or some other greyscale value depending on transparency).

Some thoughts I'm experimenting with:

Perhaps nothing more than traversing the out array and omitting those positions that are the background colour?
Trim the image to remove the background edges
Thumbnail with crop attention and then calculate

I'd appreciate any suggestions on how to exclude the background colour from the dominant colour calculation. Or more specifically, calculate the dominant colour of an image with and alpha band.

jcupitt · 2023-09-28T11:00:13Z

jcupitt
Sep 28, 2023
Maintainer

Hi @kipcole9,

I think I would do this in two passes:

Take just the alpha and count the number of 0 pixels.
Flatten the RGBA image with 0 as the background colour and count significant colours as before
Subtract the number of alpha zeros from the bin at (0, 0, 0), since all the alpha 0 pixels will have gone into that bin
Find the max position

0 replies

kipcole9 · 2023-12-14T05:03:46Z

kipcole9
Dec 14, 2023
Author

@jcupitt, I've finally got this back to the top of my list (as always, thanks for your guidance). I'm good on all the steps above I think, except:

Subtract the number of alpha zeros from the bin at (0, 0, 0), since all the alpha 0 pixels will have gone into that bin

Find the max position

I don't know how to perform step 3 in such a way that I can then call maxpos on the final histogram to derive the dominant colour.

What would you suggest?

Many thanks, --Kip

2 replies

jcupitt Dec 14, 2023
Maintainer

Histograms are just 256x1 pixel images, so you can do it in the usual way. Perhaps:

number_of_zero_alpha = (image[3] == 0).avg() * image.width * image.height / 255
histogram = image.flatten().hist_find_ndim()
zero_pixel = histogram.getpoint(0, 0)
adjusted_count = zero_pixel[0] - number_of_zero_alpha
new_zero_pixel = [adjusted_count] + zero_pixel[1:]
histogram = histogram.insert(pyvips.Image.black(1, 1) + new_zero_pixel, 0, 0)

(untested!)

kipcole9 Dec 14, 2023
Author

Thanks John. Hmmm I think I'm a little more comfortable composing rather than mutating so thanks for pointing that out.

Really appreciate your support.

akash-akya · 2023-12-14T07:17:41Z

akash-akya
Dec 14, 2023

Hey @kipcole9, I faced same issue and solved something like this based on @jcupitt suggestions.

Find Dominant Color

Mix.install([
  {:kino, "~> 0.11.2"},
  {:vix, "~> 0.24.0"}
])

alias Vix.Vips.{Image, Operation, MutableOperation}
use Vix.Operator

defmodule Dominant do
  def find(img) do
    {count, _} =
      img[3]
      |> Operation.relational_const!(:VIPS_OPERATION_RELATIONAL_EQUAL, [0])
      |> Operation.hist_find!()
      |> Operation.max!()

    black_pixels = round(count)

    max_band_value = 256
    bins = 10

    bin_size = max_band_value / bins
    midpoint = bin_size / 2

    {:ok, histogram} =
      img
      |> Operation.flatten!()
      |> Operation.hist_find_ndim(bins: bins)

    [black | rest_pixel] = Operation.getpoint!(histogram, 0, 0)

    {:ok, histogram} =
      Image.mutate(histogram, fn img ->
        pixel = [min(black - black_pixels, 0) | rest_pixel]
        MutableOperation.draw_rect!(img, pixel, 0, 0, 1, 1)
      end)

    {v, %{x: x, y: y}} = Operation.max!(histogram)
    pixel = Operation.getpoint!(histogram, x, y)
    z = Enum.find_index(pixel, &(&1 == v))

    r = x * bin_size + midpoint
    g = y * bin_size + midpoint
    b = z * bin_size + midpoint
    [round(r), round(g), round(b)]
  end
end

path = "~/Downloads/netflix.png"
{:ok, img} = Image.new_from_file(path)
img = Operation.resize!(img, 0.2)

v = Dominant.find(img)
IO.inspect(v, charlists: :as_lists)

{:ok, color} = Image.new_from_image(img, v)
color

3 replies

jcupitt Dec 14, 2023
Maintainer

Oh, nice!

mmower Jun 26, 2024

As a jumping off point I am trying to run this example locally and getting an error:

** (ArgumentError) Invalid band requested. Found 3
    (vix 0.27.0) lib/vix/vips/image.ex:157: Vix.Vips.Image.fetch/2
    (elixir 1.17.1) lib/access.ex:322: Access.get/3
    #cell:54nt3sxtib2yh74b:7: Dominant.find/1
    #cell:54nt3sxtib2yh74b:47: (file)

It seems to be img[3] which is causing the problem. Obviously I don't have the 'netflix.png' in the example, is there something specific about the image that makes this work?

I've tried with a number of random images I have here, and there always seems to be img[0], img[1], & img[2], but no img[3]

Okay I am wondering if these are red, green, & blue channels (and, if so, are these the 'bands' that I see mentioned in various places)? If so, then perhaps this code is expecting an image with an alpha channel which these images don't have.

Right that seems to be it. I tried it with an image with alpha and the code works and I think this proves my assertion that each of these images is a color channel.

akash-akya Jun 26, 2024

Obviously I don't have the 'netflix.png' in the example, is there something specific about the image that makes this work?

This example is specifically to handle alpha channel (image with 4 color channels RGBA). It seems you only have RGB colors in your case, if that is the case then code would much simpler.

Check this in Image lib for images with 3 channels

kipcole9 · 2023-12-14T07:43:30Z

kipcole9
Dec 14, 2023
Author

@akash-akya, super helpful, thank you. I figured mutating the histogram was going to be the most likely path.

0 replies

jcupitt · 2024-06-26T10:24:01Z

jcupitt
Jun 26, 2024
Maintainer

I had a "duh, of course" moment yesterday.

One of the problems with using max on the result of hist_find_ndim is that the blue channel is mapped onto bands. max will only search for maximum (x, y) positions, so if you have two maxima in the blue, it'll only find one of them.

The simple fix is to use bandunfold to unfold bands into the x dimension. This takes a 3x3 RGB image like this:

RGB RGB RGB
RGB RGB RGB
RGB RGB RGB

And unfolds it to a 9x3 mono image like this:

R G B R G B R G B 
R G B R G B R G B 
R G B R G B R G B

Now you can run max on the mono image and it won't miss anything. Use x % 3 and x // 3 to turn the x position back into a bin number.

Sample code:

#!/usr/bin/env python3

import sys
import pyvips

# number of bins in each axis 
BINS = 16
BIN_SIZE = 256 / BINS
  
image = pyvips.Image.new_from_file(sys.argv[1], access="sequential")
hist = image.hist_find_ndim(bins=BINS).bandunfold()
_, result = hist.max(size=26, out_array=True, x_array=True, y_array=True)
    
print(f"26 most common colours:")
for v, x, y in zip(result["out_array"], result["x_array"], result["y_array"]):
    r = BIN_SIZE / 2 + BIN_SIZE * (x // BINS)
    g = BIN_SIZE / 2 + BIN_SIZE * y
    b = BIN_SIZE / 2 + BIN_SIZE * (x % BINS)
    
    print(f"count = {v}, rgb = {r}, {g}, {b}")

2 replies

mmower Jun 26, 2024

Thank you. With help I've ported this code to Vix. I'm not sure where but I think you mentioned some potential challenges with the RGB colour space and while it does work for some images for others it's all very dark & muted. I think it was @akash-akya who mentioned using the LAB colour space but my naive conversion didn't change the output.

defmodule ColorSampler do
  alias Vix.Vips.Image, as: VImage
  
  def sample(%VImage{} = image) do
    bins = 16
    bin_size = round(256/bins)

    {:ok, hist} = Vix.Vips.Operation.hist_find_ndim(image, bins: bin_size)
    {:ok, bands} = Vix.Vips.Operation.bandunfold(hist)
    {:ok, mem_img} = Vix.Vips.Image.copy_memory(bands)
    {:ok, {_, arrays}} = Vix.Vips.Operation.max(mem_img, size: 26, output_array: true, x_array: true, y_array: true)
    %{"out-array": oa, "x-array": xa, "y-array": ya} = arrays

    [oa, xa, ya]
    |> Enum.zip()
    |> Enum.map(fn {_, x, y} ->
      r = (bin_size / 2) + (bin_size * div(x, bins))
      g = (bin_size / 2) + (bin_size * y)
      b = (bin_size / 2) + (bin_size * Integer.mod(x, bins))
      {round(r), round(g), round(b)}
    end)
    |> Enum.map(fn {r, g, b} -> ColorConverter.decimal_to_hex_color(r, g, b, true) end)
  end
end

kipcole9 Jun 28, 2024
Author

Just noting that for Elixir, the equivalent to Pythons // should be Integer.floor_div(x, bins)

kipcole9 · 2024-06-28T04:37:38Z

kipcole9
Jun 28, 2024
Author

John, I just wanted to add my thanks for all your efforts, and for helping on this topic. It motivated me to make some updates to my image library adding better dominant colour identification (using your current example), k-means clustering, colour reduction (using k-means clustering), colour difference (using your vips_de* functions) and colour sorting (using a kind of hlv colourspace comparison which seems to produce a reasonable result perceptually).

0 replies

Uh oh!

Dominant colour of an image with an alpha band. #3692

Uh oh!

Uh oh!

kipcole9 Sep 27, 2023

Replies: 6 comments · 7 replies

Uh oh!

jcupitt Sep 28, 2023 Maintainer

Uh oh!

kipcole9 Dec 14, 2023 Author

Uh oh!

jcupitt Dec 14, 2023 Maintainer

Uh oh!

kipcole9 Dec 14, 2023 Author

Uh oh!

akash-akya Dec 14, 2023

Find Dominant Color

Uh oh!

jcupitt Dec 14, 2023 Maintainer

Uh oh!

Uh oh!

mmower Jun 26, 2024

Uh oh!

Uh oh!

akash-akya Jun 26, 2024

Uh oh!

kipcole9 Dec 14, 2023 Author

Uh oh!

Uh oh!

jcupitt Jun 26, 2024 Maintainer

Uh oh!

mmower Jun 26, 2024

Uh oh!

kipcole9 Jun 28, 2024 Author

Uh oh!

kipcole9 Jun 28, 2024 Author

kipcole9
Sep 27, 2023

Replies: 6 comments 7 replies

jcupitt
Sep 28, 2023
Maintainer

kipcole9
Dec 14, 2023
Author

jcupitt Dec 14, 2023
Maintainer

kipcole9 Dec 14, 2023
Author

akash-akya
Dec 14, 2023

jcupitt Dec 14, 2023
Maintainer

kipcole9
Dec 14, 2023
Author

jcupitt
Jun 26, 2024
Maintainer

kipcole9 Jun 28, 2024
Author

kipcole9
Jun 28, 2024
Author