Unexpected behavior using libvips 8.13 applying filter to image and saving as AV1 #3087

johntrunc · 2022-10-09T18:33:56Z

johntrunc
Oct 9, 2022

Hi,

I have found another unexpected behavior for libvips 8.13 compared to previous 8.12 version.

For the following example I'm trying to thumbnail a jpeg and apply a filter with some alpha, after that saving the image as heif with AV1 compression.

The result with version 8.12 is the expected one (some kind of sepia filtered image) but the same code working with version 8.13 is resulting in a white image:

Result with libvips 8.12 (expected result): out-8-12-2.avif.zip
Result with libvips 8.13 (unexpected white image result): out-8-13-2.avif.zip

You can download the used example.jpeg image here if needed.

Executing the script on version 8.13 and replacing the vips_heifsave call with a vips_jpegsave is working fine so I guess is something specific of a side effect of one of the operations that I'm doing and the call to vips_heifsave function for libvips 8.13.

Not sure if this is a bug or not but it is working perfectly on version 8.12, so maybe something changed on version 8.13 that is affecting some of the operations done in the program and causing the unexpected final white image.

Thanks.

// gcc -g -Wall testing-vips.c `pkg-config vips --cflags --libs`
#include <stdio.h>
#include <vips/vips.h>

void thumbnail_avif() {
  VipsImage *in, *out, *thumbnail, *filter, *filter_matrix, *filter_matrix_alpha, *matrix;

  double matrix_array[] = {
    0.3588, 0.7044, 0.1368,
    0.2990, 0.5870, 0.1140,
    0.2392, 0.4696, 0.0912
  };

  double alpha[] = {220.0};

  if (vips_jpegload("example.jpeg", &in, "shrink", 8, "autorotate", TRUE, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_thumbnail_image(in, &thumbnail, 500, "height", 600, "size", VIPS_SIZE_BOTH, "crop", VIPS_INTERESTING_NONE, "no-rotate", TRUE, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_copy(thumbnail, &filter, NULL) == -1) {
    vips_error_exit(NULL);
  }

  matrix = vips_image_matrix_from_array(3, 3, matrix_array, 9);
  if (matrix == NULL) {
    vips_error_exit(NULL);
  }

  if (vips_recomb(filter, &filter_matrix, matrix, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_bandjoin_const(filter_matrix, &filter_matrix_alpha, alpha, 1, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_composite2(thumbnail, filter_matrix_alpha, &out, VIPS_BLEND_MODE_OVER, "x", 0, "y", 0, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_heifsave(out, "/tmp/out.avif", "Q", 75, "compression", VIPS_FOREIGN_HEIF_COMPRESSION_AV1, "strip", TRUE, NULL) == -1) {
    vips_error_exit(NULL);
  }

  g_object_unref(in);
  g_object_unref(out);
  g_object_unref(thumbnail);
  g_object_unref(filter);
  g_object_unref(filter_matrix);
  g_object_unref(filter_matrix_alpha);
  g_object_unref(matrix);
}

int main(int argc, char **argv)
{
  if (VIPS_INIT("testing-vips")) {
    vips_error_exit(NULL);
  }

  thumbnail_avif();

  return(0);
}

lovell · 2022-10-09T20:28:20Z

lovell
Oct 9, 2022
Maintainer

Did you see the new bitdepth parameter added in 8.13.0? Depending on the software you're using to view the output, you may want to set this to 8 (i.e. not high bit rate) to ensure compatibility.

https://www.libvips.org/API/current/VipsForeignSave.html#vips-heifsave

1 reply

johntrunc Oct 10, 2022
Author

I have modified the call to vips_heifsave using bitdepth with value 8 and still generating an empty white avif image (using Google Chrome to see the result):

// gcc -g -Wall testing-vips.c `pkg-config vips --cflags --libs`
#include <stdio.h>
#include <vips/vips.h>

void thumbnail_avif() {
  VipsImage *in, *out, *thumbnail, *filter, *filter_matrix, *filter_matrix_alpha, *matrix;

  double matrix_array[] = {
    0.3588, 0.7044, 0.1368,
    0.2990, 0.5870, 0.1140,
    0.2392, 0.4696, 0.0912
  };

  double alpha[] = {220.0};

  if (vips_jpegload("example.jpeg", &in, "shrink", 8, "autorotate", TRUE, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_thumbnail_image(in, &thumbnail, 500, "height", 600, "size", VIPS_SIZE_BOTH, "crop", VIPS_INTERESTING_NONE, "no-rotate", TRUE, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_copy(thumbnail, &filter, NULL) == -1) {
    vips_error_exit(NULL);
  }

  matrix = vips_image_matrix_from_array(3, 3, matrix_array, 9);
  if (matrix == NULL) {
    vips_error_exit(NULL);
  }

  if (vips_recomb(filter, &filter_matrix, matrix, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_bandjoin_const(filter_matrix, &filter_matrix_alpha, alpha, 1, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_composite2(thumbnail, filter_matrix_alpha, &out, VIPS_BLEND_MODE_OVER, "x", 0, "y", 0, NULL) == -1) {
    vips_error_exit(NULL);
  }

  if (vips_heifsave(out, "/tmp/out.avif", "bitdepth", 8, "Q", 75, "compression", VIPS_FOREIGN_HEIF_COMPRESSION_AV1, "strip", TRUE, NULL) == -1) {
    vips_error_exit(NULL);
  }

  g_object_unref(in);
  g_object_unref(out);
  g_object_unref(thumbnail);
  g_object_unref(filter);
  g_object_unref(filter_matrix);
  g_object_unref(filter_matrix_alpha);
  g_object_unref(matrix);
}

int main(int argc, char **argv)
{
  if (VIPS_INIT("testing-vips")) {
    vips_error_exit(NULL);
  }

  thumbnail_avif();

  return(0);
}

jcupitt · 2022-10-10T15:04:17Z

jcupitt
Oct 10, 2022
Maintainer

Hi, yes heifsave now supports HDR, so your float image is being saved as an HDR AVIF. Float images use 0 - 1 for black to white, so your 0 255 image is just white.

As a workaround, you can add this just after the recomb:

{
        VipsImage *x;

        if (vips_cast_uchar(filter_matrix, &x, NULL))
            vips_error_exit(NULL);
        g_object_unref (filter_matrix);
        filter_matrix = x;
}

heifsave should probably not save all float images as HDR. Maybe we should only enable this for images tagged as scRGB (which do use 0-1 for black to white)?

2 replies

jfcalvo Oct 10, 2022

It is possible to define the matrix_array in a different way instead of doing the vips_cast_uchar? (I'm having a similar problem)

jcupitt Oct 10, 2022
Maintainer

recomb always outputs float, so you need to cast after that if you want to be 8 bits.

Looking at the code I think this is an error in the new 16-bit HEIF path. I'll try to fix this for the next 8.13 release..

jcupitt · 2022-10-11T07:47:24Z

jcupitt
Oct 11, 2022
Maintainer

I improved the rules it uses for 16-bit heifsave: fbef674 These now parallel the ones it uses for PNG save, so it should be more consistent.

This will be in 8.13.3. Thanks for reporting this @johntrunc !

7 replies

jcupitt Nov 7, 2022
Maintainer

It's working for me, I ran:

$ gcc -g -Wall trunc.c `pkg-config vips --cflags --libs`
$ ./a.out ~/pics/example.jpg x.avif
$ vips copy x.avif x.png

To make:

I tried with 8.13 and 8.14 (git master libvips).

That's with my version of your code, I'll try your original.

jcupitt Nov 7, 2022
Maintainer

Yes, your code works too. I'm on ubuntu 22.04, with their standard libheif 1.12.0, libaom 3.3.0 packages.

johntrunc Nov 13, 2022
Author

My fault, I missed to add the call to vips_cast_uchar after vips_recomb` and now it's working fine.

One last question @jcupitt. Why is the vips_cast_uchar not necessary when saving the output as PNG? (But if missing when saving the output as AVIF is generating those green artifacts).

Thanks for taking the time to help me with this.

jcupitt Nov 13, 2022
Maintainer

8.13 has different rules for AVIF -- it tries to save float images as HDR.

8.14 fixes this and PNG and AVIF use the same rules, so the extra cast won't be necessary.

johntrunc Nov 13, 2022
Author

I take note. Thank you very much @jcupitt and @lovell for your help.

jcupitt · 2022-10-11T07:59:01Z

jcupitt
Oct 11, 2022
Maintainer

You probably know, but your test prog could be simplified a little.

These lines:

  if (vips_jpegload("example.jpeg", &in,
  if (vips_thumbnail_image(in, &thumbnail, 500,

Would be better as:

  if (vips_thumbnail("example.jpeg", &thumbnail, 500, ...

You'll see better performance and lower memory use from doing the autorotate later.

You can skip the copy afterwards.

This line:

  double matrix_array[] = {
    0.3588, 0.7044, 0.1368,
    0.2990, 0.5870, 0.1140,
    0.2392, 0.4696, 0.0912
  };
  matrix = vips_image_matrix_from_array(3, 3, matrix_array, 9);

Could be:

  matrix = vips_image_new_matrixv(3, 3,  {
    0.3588, 0.7044, 0.1368,
    0.2990, 0.5870, 0.1140,
    0.2392, 0.4696, 0.0912
  });

Instead of:

  double alpha[] = {220.0};
  if (vips_bandjoin_const(filter_matrix, &filter_matrix_alpha, alpha, 1, NULL)...

You could write:

  if (vips_bandjoin_const1(filter_matrix, &filter_matrix_alpha, 220, NULL)...

2 replies

jcupitt Oct 11, 2022
Maintainer

I would also chain calls with ||, and use a scope object. Perhaps:

// gcc -g -Wall testing-vips.c `pkg-config vips --cflags --libs`
#include <stdio.h>
#include <vips/vips.h>

int thumbnail_avif(VipsObject *scope,
        const char *input_filename, const char *output_filename)
{
        VipsImage **t = (VipsImage **) vips_object_local_array(scope, 10);

        if (
                vips_thumbnail(input_filename, &t[0], 500,
                        "height", 600,
                        "size", VIPS_SIZE_BOTH,
                        "crop", VIPS_INTERESTING_NONE,
                        NULL) ||
                !(t[1] = vips_image_new_matrixv(3, 3,
                        0.3588, 0.7044, 0.1368,
                        0.2990, 0.5870, 0.1140,
                        0.2392, 0.4696, 0.0912)) ||
                vips_recomb(t[0], &t[2], t[1], NULL) ||
                vips_cast_uchar(t[2], &t[3], NULL) ||
                vips_bandjoin_const1(t[3], &t[4], 220, NULL) ||
                vips_composite2(t[0], t[4], &t[5], 
                        VIPS_BLEND_MODE_OVER, NULL) ||
                vips_heifsave(t[5], output_filename,
                        "bitdepth", 8,
                        "Q", 75, 
                        "compression", VIPS_FOREIGN_HEIF_COMPRESSION_AV1,
                        "strip", TRUE,
                        NULL))
                return -1;

        return 0;
}

int main(int argc, char **argv)
{
  VipsObject *scope;

  if (VIPS_INIT(argv[0]))
    vips_error_exit(NULL);

  scope = VIPS_OBJECT(vips_image_new());
  if (thumbnail_avif(scope, argv[1], argv[2])) {
          g_object_unref (scope);
          vips_error_exit (NULL);
  }
  g_object_unref (scope);

  return 0;
}

johntrunc Oct 11, 2022
Author

Thank you very much for the suggestions. I need to take a look to the documentation because the scope object as example is totally new to me!.

Uh oh!

Unexpected behavior using libvips 8.13 applying filter to image and saving as AV1 #3087

Uh oh!

johntrunc Oct 9, 2022

Replies: 4 comments · 12 replies

Uh oh!

lovell Oct 9, 2022 Maintainer

Uh oh!

Uh oh!

johntrunc Oct 10, 2022 Author

Uh oh!

jcupitt Oct 10, 2022 Maintainer

Uh oh!

Uh oh!

jfcalvo Oct 10, 2022

Uh oh!

jcupitt Oct 10, 2022 Maintainer

Uh oh!

jcupitt Oct 11, 2022 Maintainer

Uh oh!

Uh oh!

jcupitt Nov 7, 2022 Maintainer

Uh oh!

jcupitt Nov 7, 2022 Maintainer

Uh oh!

johntrunc Nov 13, 2022 Author

Uh oh!

jcupitt Nov 13, 2022 Maintainer

Uh oh!

johntrunc Nov 13, 2022 Author

Uh oh!

Uh oh!

jcupitt Oct 11, 2022 Maintainer

Uh oh!

Uh oh!

jcupitt Oct 11, 2022 Maintainer

Uh oh!

johntrunc Oct 11, 2022 Author

johntrunc
Oct 9, 2022

Replies: 4 comments 12 replies

lovell
Oct 9, 2022
Maintainer

johntrunc Oct 10, 2022
Author

jcupitt
Oct 10, 2022
Maintainer

jcupitt Oct 10, 2022
Maintainer

jcupitt
Oct 11, 2022
Maintainer

jcupitt Nov 7, 2022
Maintainer

jcupitt Nov 7, 2022
Maintainer

johntrunc Nov 13, 2022
Author

jcupitt Nov 13, 2022
Maintainer

johntrunc Nov 13, 2022
Author

jcupitt
Oct 11, 2022
Maintainer

jcupitt Oct 11, 2022
Maintainer

johntrunc Oct 11, 2022
Author