Letterboxing Images and Iterating VipsImage pixel data #2261

Transigent · 2021-05-23T12:19:39Z

Transigent
May 23, 2021

Hi and sorry about the noobness of this question, I am fairly new to C and VIPS.

I am successfully using libvips in a VS2017 C++ project to load thumbnails from files on disk, using the C API. Its so FAST compared to the existing opencv implementation! Thanks for the great library.

My questions are several:

1) Existing code requires 640 x 640 thumbnails, I am currently using code like this:
vips_thumbnail(FileName, &out, 640)
which gets me nearly what I need except that only the longest axis is 640, I need to supply the image data letterboxed (or pillarboxed) ie. so that it fills the whole 640 x 640 but centered and maintaining aspect ratio. I saw a weserv enhancement discussing this weserv/images#80 (comment) but I couldn't find if this has been implemented in libvips or in the thumbnailer. If not is there another fast way?

2) Theres probably an obvious answer but I could not find how to directly access the image data in the VipsImage object. The documentation description of VipsImage states that "VIPS images are three-dimensional arrays, the dimensions being width, height and bands" but I cant find examples that show how to iterate through the data, the closest I could find is a region example here which I dont know how to transmogrify so it works with an VipsImage. Is there an example of this somewhere?

3) I want to use libvips to do what an existing block of opencv code does using a cv::Mat. It iterates through the (BGR) pixel data to build a new array organized like this :

All R pixel data, followed by all G pixel data, then B pixel data.
Pixel values are changed (from uchar) to floats with values from 0 to 1
(It is formatted like this for feeding to a neural network)

At first I thought I would do the transformation manually like the original code but I have seen some really powerful code snippets like this and it occurs that libvips might already have the tools to reformat the data the way I need it. Is there any inbuilt functionality to do that?

Thanks for any advice!

jcupitt · 2021-05-23T13:18:45Z

jcupitt
May 23, 2021
Maintainer

Hello @Transigent,

libvips has efficient operation compositing -- you can run one operation after another, and behind the scenes they join together into a fast and low-memory-use combined operation. So the idea behind the design of the API is that operations only implement the minimum functionality, and you get complex effects by combining primitives.

thumbnail shrinks to a bounding box. Use gravity to extend an image within a new canvas, so you need (in python, for brevity):

thumb = pyvips.Image.thumbnail(filename, 640)
boxed = thumb.gravity("centre", 640, 640, extend="white")

You can't really iterate over pixels in libvips, since images don't really exist. Everything is a delayed computation and pixels are only created on demand. You can either implement a new vips operation (needs good C or C++), or render the whole image to an area of memory and then treat it like any other array. Have a look at vips_image_write_to_memory().

You can split your image up like this (in Python, again):

# rescale to float 0 - 1
image = image / 255.0
# split bands to an array of images
bands = image.bandsplit()
# join those images vertically
image = pyvips.Image.arrayjoin(bands, across=1)

Then write to memory to make a huge float array.

You'll find python is as fast as C for libvips, so it might be more convenient.

1 reply

Transigent May 24, 2021
Author

Thanks so much @jcupitt for the super fast response. That explains a few things and makes a lot of sense now.

Consider this solved.

Transigent · 2021-05-25T08:56:08Z

Transigent
May 25, 2021
Author

OK I thought I had it under control but it seems I am doing something wrong.

I set up a basic test harness to try this out. Here is my code so far:

int ShrinkImage(char* FileName) {

	int result;
	VipsImage *thum = new VipsImage;
	VipsImage *boxed = new VipsImage;

	if (vips_thumbnail(FileName, &thum, 640))
		vips_error_exit(NULL);


//	if (vips_gravity(thum, &boxed, VIPS_COMPASS_DIRECTION_CENTRE,  640, 640, "extend", VIPS_EXTEND_BLACK)) 
//		vips_error_exit(NULL);


	if (vips_image_write_to_file(thum, "C:\\Users\\Chris\\Desktop\\TestImageOut.jpg", "Q", 70, "strip", TRUE, NULL))
		vips_error_exit(NULL);


	//g_object_unref(thum);
	//g_object_unref(boxed);
	
	thum = NULL;
	boxed = NULL;

	return(0);
}

It is being called from this method:

int ImageShrinkTest() {

	if (VIPS_INIT("8"))
		vips_error_exit(NULL);

	char str[] = "C:/Users/Chris/Desktop/ALL Test Images/Big Images/a10000.jpg";
	char str2[] = "C:/Users/Chris/Desktop/ALL Test Images/Big Images/a.jpg";
	char str3[] = "C:/Users/Chris/Desktop/ALL Test Images/Big Images/7360.jpg";
 
	ShrinkImage(str);
	ShrinkImage(str2);
	ShrinkImage(str3);
	
	return(0);
}

I started from the code sample at the bottom of the C API documentation page and translated from your python sample.

This code works fine until I reenable the section:

	if (vips_gravity(thum, &boxed, VIPS_COMPASS_DIRECTION_CENTRE,  640, 640, "extend", VIPS_EXTEND_BLACK)) 
		vips_error_exit(NULL);

at which point I get an exception on the vips_gravity line:

Exception thrown at 0x00007FF8103ED090 (msvcrt.dll) in test.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.

If you are wondering why I didn't just leave it as:

	VipsImage *thum;
	VipsImage *boxed;

its because it only gets one pass through ShrinkImage then hangs at

if (vips_thumbnail(FileName, &thum, 640))
		vips_error_exit(NULL);

And if I used

	g_object_unref(thum);
	g_object_unref(boxed);

like in the C sample, it hangs on return(0) with

Exception thrown at 0x0000000067F3132D (libgobject-2.0-0.dll) in test.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.

I must be doing something wrong, if you have any thoughts they would be greatly welcome.

0 replies

jcupitt · 2021-05-25T09:11:37Z

jcupitt
May 25, 2021
Maintainer

You're almost there -- you need NULLs at the end of arg lists in C, and your reference counting it not quite right. I made you a small sample program:

/* compile with
 *
 *      gcc -g -Wall try325.c `pkg-config vips --cflags --libs`
 */

#include <vips/vips.h>

int
main( int argc, char **argv )
{       
        VipsImage *image;
        VipsImage *x;

        if (VIPS_INIT (argv[0]))
                vips_error_exit (NULL);

        if (vips_thumbnail (argv[1], &image, 640, NULL))
                vips_error_exit (NULL);

        if (vips_gravity (image, &x, VIPS_COMPASS_DIRECTION_CENTRE, 640, 640,
                "extend", VIPS_EXTEND_BLACK,
                NULL)) {
                g_object_unref (image);
                vips_error_exit (NULL);
        }
        g_object_unref (image);
        image = x;

        if (vips_image_write_to_file (image, argv[2],
                "Q", 70,
                "strip", TRUE, 
                NULL)) {
                g_object_unref (image);
                vips_error_exit (NULL);
        }

        g_object_unref (image);

        return 0;
}

1 reply

Transigent May 25, 2021
Author

Geez you code fast, thanks so much again. I'll give that a try.

jcupitt · 2021-05-25T09:23:04Z

jcupitt
May 25, 2021
Maintainer

Ooop, pressed ctrl-enter by mistake and posted too early.

NULLs to terminate arg lists

C functions aren't told how many argument they are passed, so you need to mark the end of the argument list with a NULL. gcc will warn if you forget, perhaps MSVC has some warning you can enable for this?

Reference counting

The C API creates the output image for you, so the pattern is always:

VipsImage *input-image1 = some-code;
VipsImage *input-image2 = some-code;
VipsImage *output-image;

if (vips_xxxxx(input-image1, input-image2, &output-image, optional-args, NULL))
   error-handling

The vips_xxx() functions always take refs to the inputs, so you can drop your input refs afterwards if you no longer need them. The new image that is returned has one ref which you own and it's your responsibility to unref it when you're done with it.

That example above uses a single image variable pointing to the head of the pipeline, and moves it along as new nodes are added. libvips also supports reference pools which can be less work for complex cases.

libvips C++ binding

The libvips C++ API does not use varargs, so it's a bit simpler. It might be worth looking at this -- it'll automate ref counting too.

It's slightly awkward to set up under Windows. The problem is that C++ does not have a ABI, so you must use exactly the same C++ compiler for your whole project. This means the libvips C++ win binary (built with g++) won't work with MSVC C++.

You need to copy the libvips C++ API source code into your own project and build it with your own code. It's just a few files and pretty simple to incorporate:

https://github.com/libvips/libvips/tree/master/cplusplus

The C++ API source is notionally LGPL since it's shipped with the rest of libvips, but we let people do anything they like with it because of this problem. Perhaps it should be broken out into a separate MIT-licenced project.

0 replies

jcupitt · 2021-05-25T09:32:40Z

jcupitt
May 25, 2021
Maintainer

In C++, that test prog would be:

/* compile with
 *
 *      g++ -g -Wall try325.cpp `pkg-config vips-cpp --cflags --libs`
 */

#include <iostream>
#include <vips/vips8>

using namespace vips;
using namespace std;

int
main( int argc, char **argv )
{
        if (VIPS_INIT (argv[0]))
                vips_error_exit (NULL);

        try {
                VImage image = VImage::thumbnail(argv[1], 640);
                image = image.gravity(VIPS_COMPASS_DIRECTION_CENTRE, 640, 640, VImage::option()       
                        ->set("extend", "black"));
                image.write_to_file(argv[2], VImage::option()
                        ->set("strip", TRUE)
                        ->set("Q", 70));
        }
        catch (const VError &err ) {
                cout << err.what();
        }

        return 0;
}

So no annoying NULLs, and ref counting is done for you.

3 replies

Transigent May 27, 2021
Author

Once again thank @jcupitt for the great explanation. You went way above and beyond to explain the Null termination and reference counting concepts to a beginner. Thanks so much for the information. I am now using those functions successfully.

I will try to build the C++ files into the project over the weekend.

With regard to your earlier Python sample:

# rescale to float 0 - 1
image = image / 255.0
# split bands to an array of images
bands = image.bandsplit()
# join those images vertically
image = pyvips.Image.arrayjoin(bands, across=1)

I suppose there is no equivalent in C that is as elegant? I found vips_extract_band () and friends but could not find vips_bandsplit() in the C implementation of libvips, this might be a Python only thing.

Thanks once again.

jcupitt May 27, 2021
Maintainer

The C++ API is pretty close -- it has bandsplit and operator overloading.

You could write:

VImage rescaled = image / 255.0;
std::vector<VImage> bands = rescaled.bandsplit();
VImage result = VImage::arrayjoin(bands, VImage::option()->set("across", 1));

Transigent May 30, 2021
Author

Thanks again @jcupitt, for the support, that sounds great. I am very keen to try the C++ API.

I have tried a few times to create a basic project which uses the C++ files as discussed, and which also compiles, but I must be doing something wrong, I get 180 errors which seems to imply I am missing some required include or something.

My steps so far are:

Create a C++ console app in VS2017, set to Debug and x64
Extract the 'vips-dev-8.10' folder from vips-dev-w64-all-8.10.6.zip to where the project file is
Extract 'cplusplus' folder from 'libvips-8.10.6.zip' to where the project file is
Add the following folders to Project > Properties > C/C++->General > Additional Include Directories

C:\Users\Chris\source\repos\ConsoleApplicationVIPS3\ConsoleApplicationVIPS3\cplusplus\include\vips
C:\Users\Chris\source\repos\ConsoleApplicationVIPS3\ConsoleApplicationVIPS3\vips-dev-8.10\include
C:\Users\Chris\source\repos\ConsoleApplicationVIPS\ConsoleApplicationVIPS\vips-dev-8.10\lib\glib-2.0\include
C:\Users\Chris\source\repos\ConsoleApplicationVIPS\ConsoleApplicationVIPS\vips-dev-8.10\include\glib-2.0

Add the 5 .cpp files from the cplusplus folder into the project Source Files folder.
Build the project.

As a result I get the following errors:

I'm sorry for all the trouble. Any advice would be much appreciated.

Transigent · 2021-06-03T04:20:59Z

Transigent
Jun 3, 2021
Author

Sorry I wasn't sure if you got the previous message.
If this question is better asked in a C++ forum let me know.

2 replies

jcupitt Jun 3, 2021
Maintainer

I didn't answer because I didn't know, sorry. I've not done it myself, but I know that's what eg. the node bindings do on Windows.

Transigent Jun 3, 2021
Author

OK thanks I will keep looking

Uh oh!

Letterboxing Images and Iterating VipsImage pixel data #2261

Uh oh!

Uh oh!

Transigent May 23, 2021

Replies: 6 comments · 7 replies

Uh oh!

Uh oh!

jcupitt May 23, 2021 Maintainer

Uh oh!

Transigent May 24, 2021 Author

Uh oh!

Transigent May 25, 2021 Author

Uh oh!

jcupitt May 25, 2021 Maintainer

Uh oh!

Transigent May 25, 2021 Author

Uh oh!

Uh oh!

jcupitt May 25, 2021 Maintainer

NULLs to terminate arg lists

Reference counting

libvips C++ binding

Uh oh!

jcupitt May 25, 2021 Maintainer

Uh oh!

Transigent May 27, 2021 Author

Uh oh!

jcupitt May 27, 2021 Maintainer

Uh oh!

Uh oh!

Transigent May 30, 2021 Author

Uh oh!

Transigent Jun 3, 2021 Author

Uh oh!

jcupitt Jun 3, 2021 Maintainer

Uh oh!

Transigent Jun 3, 2021 Author

Transigent
May 23, 2021

Replies: 6 comments 7 replies

jcupitt
May 23, 2021
Maintainer

Transigent May 24, 2021
Author

Transigent
May 25, 2021
Author

jcupitt
May 25, 2021
Maintainer

Transigent May 25, 2021
Author

jcupitt
May 25, 2021
Maintainer

jcupitt
May 25, 2021
Maintainer

Transigent May 27, 2021
Author

jcupitt May 27, 2021
Maintainer

Transigent May 30, 2021
Author

Transigent
Jun 3, 2021
Author

jcupitt Jun 3, 2021
Maintainer

Transigent Jun 3, 2021
Author