BMP does not work in very specific situations ("bmp is not a known buffer format") #4035

L3tum · 2024-07-09T14:27:04Z

L3tum
Jul 9, 2024

Hey, I realize this will probably be stretching your willingness to help because the whole setup is very custom-made, but I'm asking in the hopes that you've had a similar issue in the past and could give me a pointer to what may fix it.

So here's the issue in as much detail as possible: I've adopted the build script @kleisauke uses for the NetVips distribution to also include imagemagick, because I need BMP support (please don't ask why). I've done it pretty straightforward with the following additions to the script (beyond some version handling boilerplate):

# ImageMagick can make use of freetype apparently so it's after that step
mkdir ${DEPS}/im
$CURL https://github.com/ImageMagick/ImageMagick/archive/refs/tags/${VERSION_IMAGEMAGICK}.tar.gz | tar xzC ${DEPS}/im --strip-components=1
cd ${DEPS}/im
./configure --host=${CHOST} --prefix=${TARGET} --disable-shared --enable-static --without-magick-plus-plus --without-perl
make install-strip
....
[libvips build command]
-Dmagick=enabled -Dmodules=disabled

This produces, as far as I can see, a static library for imagemagick, as well as an ImageMagick folder which contains some documentation. Because libvips should be statically built, and ldd libvips.so doesn't list any dependencies beyond the usual.
If I use Debain as base for my distribution, then everything works as expected. However, if I use the Scratch docker image and copy libvips.so (and its dependencies) over (the same as I do when using debian), then everything works, except any format using libmagick (TIFF does work but from what I can tell, you've added code for it to work without imagemagick). This leads me to believe that something is missing from the produced libvips.so (and dependencies), but gets installed with Debian. But for all the documentation available, there should be nothing missing.

Here's the ldd for libvips:

> ldd libvips.so.42
        linux-vdso.so.1 (0x00007ffed23a7000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f6db55e9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6db550a000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6db54ea000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6db5309000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6db6f57000)

When executing on the Debian distribution, here's what libvips seems to open:

openat(AT_FDCWD, "/usr/local/lib/libvips.so.42", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/usr/local/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/usr/local/lib/libgomp.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgomp.so.1", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/sys/devices/system/cpu/possible", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 7
[...]
[immediately before creating a BMP image]
openat(AT_FDCWD, "/target/share/ImageMagick-7/policy.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/lib/ImageMagick-7.1.1//config-Q16HDRI/policy.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/etc/ImageMagick-7/policy.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/share/doc/ImageMagick-7/policy.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/root/.config/ImageMagick/policy.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/share/ImageMagick-7/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/lib/ImageMagick-7.1.1//config-Q16HDRI/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/etc/ImageMagick-7/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/share/doc/ImageMagick-7/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/root/.config/ImageMagick/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/share/ImageMagick-7/english.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/lib/ImageMagick-7.1.1//config-Q16HDRI/english.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/etc/ImageMagick-7/english.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/share/doc/ImageMagick-7/english.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/root/.config/ImageMagick/english.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/share/ImageMagick-7/log.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/lib/ImageMagick-7.1.1//config-Q16HDRI/log.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/etc/ImageMagick-7/log.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/target/share/doc/ImageMagick-7/log.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/root/.config/ImageMagick/log.xml", O_RDONLY) = -1 ENOENT (No such file or directory)

Do notice that there's nothing actually opened in the second part of it. The only thing missing from ldd libvips.so is libgomp.so, but the file seems to be present regardless.
I have no idea what else to check. I'm running a complete trace right now to see if there's anything funky going on aside from simple dlopen calls, but I really doubt it. I can't run a trace on the Scratch image by extension of it being scratch and not containing anything I could use. The one thing I could check, if all the files it opens in the Debain version are present, does seem to be the case.
The BMP file is actually created and is a valid BMP file containing the image I expect, so it doesn't fail silently or anything like that.

kleisauke · 2024-07-09T17:01:14Z

kleisauke
Jul 9, 2024
Maintainer

Does the build output indicate that libvips was successfully built against IM? You should see something similar to the following:

    selected Magick package           : MagickCore (dynamic module: NO)
    Magick API version                : magick7
    Magick load                       : YES
    Magick save                       : YES

You might also try passing --without-modules to the IM build, although this should be enabled automatically if you use if you use --disable-shared. Another option is to pass --disable-installed --enable-zero-configuration to build a self-contained, embeddable, zero-configuration IM binary.

Note that it's generally much easier to install libvips via your package manager (see e.g. #1184). I usually recommend using Rocky Linux (or any other RHEL derivative) in combination with Remi's RPM repository, a third-party repository that includes an up-to-date version of libvips. I've provided an example with BMP support via GraphicsMagick here:
kleisauke/net-vips#223 (comment)
(for IM7 or IM6, simply replace vips-magick-gm with vips-magick-im7 or vips-magick-im6, respectively)

3 replies

L3tum Jul 10, 2024
Author

Hey, thank you for replying!

--- Before reading the rest, I just thought the difference may be that my tests on the Debian installation use a local file, while my non-debian installation uses remote files? Both use a Source as input though. The error would be pretty weird if that's the case. But that's the only diffeence I can think of, beyond the obvious.

libvips seems to pick it up

    selected Magick package           : MagickCore (dynamic module: NO)
    Magick API version                : magick7
    Magick load                       : YES
    Magick save                       : YES

Here's the ImageMagick compilation configuration output:

ImageMagick 7.1.1-34 is configured as follows. Verify that this configuration
matches your expectations.
Host system type: x86_64-pc-linux-gnu
Build system type: x86_64-pc-linux-gnu
                    Option				Value
------------------------------------------------------------------------------
Shared libraries    --enable-shared=no			no
Static libraries    --enable-static=yes			yes
Build utilities     --with-utilities=yes		yes
Module support      --with-modules=no			no
GNU ld              --with-gnu-ld=yes			yes
OpenMP              --enable-openmp=			yes
Security policy     --with-security-policy=open	        open
Quantum depth       --with-quantum-depth=16		16
Channel mask depth  --enable-64bit-channel-masks=no	32
High Dynamic Range Imagery
                    --enable-hdri=yes			yes
Install documentation:					yes
Memory allocation library:
  JEMalloc          --with-jemalloc=no			no
  MTMalloc          --with-mtmalloc=no			no
  TCMalloc          --with-tcmalloc=no			no
  UMem              --with-umem=no			no
Delegate library configuration:
  BZLIB             --with-bzlib=yes			no
  Autotrace         --with-autotrace=no			no
  DJVU              --with-djvu=yes			no
  DPS               --with-dps=no			no
  FFTW              --with-fftw=no			no
  FLIF              --with-flif=no			no
  FlashPIX          --with-fpx=no			no
  FontConfig        --with-fontconfig=yes		yes
  FreeType          --with-freetype=yes			yes
  Ghostscript lib   --with-gslib=no			no
  Graphviz          --with-gvc=yes			no
  HEIC              --with-heic=yes			yes
  JBIG              --with-jbig=yes			no
  JPEG v1           --with-jpeg=yes			yes
  JPEG XL           --with-jxl=yes			yes
  DMR               --with-dmr=yes			no
  LCMS              --with-lcms=yes			yes
  LQR               --with-lqr=yes			no
  LTDL              --with-ltdl=no			no
  LZMA              --with-lzma=yes			no
  Magick++          --with-magick-plus-plus=no		no
  OpenEXR           --with-openexr=yes			no
  OpenJP2           --with-openjp2=yes			yes
  PANGO             --with-pango=yes			yes
  PERL              --with-perl=no			no
  PNG               --with-png=yes			yes
  RAQM              --with-raqm=yes			no
  RAW               --with-raw=yes			no
  RSVG              --with-rsvg=no			no
  TIFF              --with-tiff=yes			yes
  UHDR              --with-uhdr=no			no
  WEBP              --with-webp=yes			yes
  WMF               --with-wmf=no			no
  X11               --with-x=				no
  XML               --with-xml=yes			yes
  ZIP               --with-zip=yes			no
  ZLIB              --with-zlib=yes			yes
  ZSTD              --with-zstd=yes			no
Delegate program configuration:
  GhostPCL          None				pcl6 (unknown)
  GhostXPS          None				gxps (unknown)
  Ghostscript       None				gs (unknown)
Font configuration:
  Apple fonts       --with-apple-font-dir=default	
  Dejavu fonts      --with-dejavu-font-dir=default	none
  Ghostscript fonts --with-gs-font-dir=default		none
  URW-base35 fonts  --with-urw-base35-font-dir=default	none
  Windows fonts     --with-windows-font-dir=default	none
X11 configuration:
  X_CFLAGS          = 
  X_PRE_LIBS        = 
  X_LIBS            = 
  X_EXTRA_LIBS      = 
Options used to compile and link:
  PREFIX            = /target
  EXEC-PREFIX       = /target
  VERSION           = 7.1.1-34
  CC                = gcc
  CFLAGS            = 
  CPPFLAGS          = -I/target/include -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_CHANNEL_MASK_DEPTH=32
  PCFLAGS           = 
  DEFS              = -DHAVE_CONFIG_H
  LDFLAGS           = -L/target/lib -Wl,--gc-sections -Wl,-rpath=$ORIGIN/
  LIBS              = -L/target/lib -llcms2 -llcms2_fast_float -llcms2_threaded -pthread -lm -L/target/lib -ltiff -lwebp -lm -L/target/lib -lwebp -lm -L/target/lib -lsharpyuv -lm -ljpeg -lz -L/target/lib -lz -lharfbuzz -lm -lfreetype -lglib-2.0 -lm -pthread -lbrotlidec -L/target/lib -lbrotlicommon -ljpeg -L/target/lib -ljxl -lm -lstdc++ -lhwy -lbrotlienc -L/target/lib -lbrotlicommon -lbrotlidec -L/target/lib -lbrotlicommon -ljxl_cms -lstdc++ -llcms2 -llcms2_fast_float -llcms2_threaded -pthread -lm -ljxl_threads -lm -L/target/lib -lpng16 -lm -lz -lm -L/target/lib -L/target/lib -lz -L/target/lib -lfontconfig -lz -lharfbuzz -lm -lfreetype -lglib-2.0 -lm -pthread -lbrotlidec -L/target/lib -lbrotlicommon -lexpat -lm -L/target/lib -lheif -lstdc++ -L/target/lib -laom -lm -lsharpyuv -lm -L/target/lib -lwebpmux -lm -L/target/lib -lwebp -lm -L/target/lib -lsharpyuv -lm -lwebpdemux -L/target/lib -lwebp -lm -L/target/lib -lsharpyuv -lm -L/target/lib -lwebp -lm -L/target/lib -lsharpyuv -lm -L/target/lib -lopenjp2 -lm -L/target/lib -lpangocairo-1.0 -lm -lm -lm -L/target/lib -lbrotlicommon -lz -lm -L/target/lib -lbrotlicommon -lm -lm -lz -L/target/lib -L/target/lib -lm -lm -L/target/lib -lbrotlicommon -lm -L/target/lib -lbrotlicommon -lpangoft2-1.0 -lpango-1.0 -lm -lm -lm -L/target/lib -lbrotlicommon -lz -lm -L/target/lib -lbrotlicommon -lm -lm -lz -L/target/lib -L/target/lib -lm -lm -L/target/lib -lbrotlicommon -lm -L/target/lib -lbrotlicommon -lm -lm -lm -L/target/lib -lbrotlicommon -lz -lm -L/target/lib -lbrotlicommon -lm -lm -lz -L/target/lib -L/target/lib -lm -lm -L/target/lib -lbrotlicommon -lm -L/target/lib -lbrotlicommon -lgio-2.0 -lgobject-2.0 -lffi -lgmodule-2.0 -lfribidi -lm -lm -lm -L/target/lib -lbrotlicommon -lz -lm -L/target/lib -lbrotlicommon -lcairo -lm -ldl -lpng16 -lm -lz -L/target/lib -L/target/lib -lfontconfig -lm -lm -L/target/lib -lbrotlicommon -lexpat -lz -lharfbuzz -lm -lfreetype -lglib-2.0 -lbrotlidec -L/target/lib -lbrotlicommon -lpixman-1 -lm -pthread -L/target/lib -lxml2 -lm -lz -L/target/lib -lz -lm -lpthread -lgomp
  CXX               = g++
  CXXFLAGS          = -march=nehalem -Os -fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -fno-asynchronous-unwind-tables -ffunction-sections -fdata-sections -pthread
  FEATURES          = Channel-masks(32-bit) Cipher DPC HDRI OpenMP
  DELEGATES         = fontconfig freetype heic jng jp2 jpeg jxl lcms pangocairo png tiff webp xml zlib

I'll try your suggestions on it today. Though I would expect libvips to still accept BMP as a format, and then fail during transcode, because even a missing dynamic library shouldn't influence VipsForeignSave, should it? Or at the very least it should show up as a failed openat/dlopen call in my trace. The weird thing is still that it doesn't fail on a regular Debian installation, but it also doesn't seem to be doing anything to suggest it should fail on a non-Debian installation.

P.S. Concerning the libvips package-manager version: I primarily started this to make use of AVIF (and a bit later, JPEG-XL), which weren't present in Debian 9-11. I'm actually susprised that libvips in Debian 12 is almost up to date. Neverthelesss, the current iteration is mostly there to eliminate the libxml/libicu dependency that seems to come with basically everything nowadays and blows up our app size from ~100MB to ~500MB due to all the translation data that we really don't need, as well as to remove HEIF support due to the licensing issues around it (yes we could leave it in, but I've already been asked 5 times to activate it "regardless of consequences" so I'd rather not take the chance 😆 )

kleisauke Jul 10, 2024
Maintainer

Ah, I was assuming you meant BMP load, but the bmp is not a known buffer format-error is cleary a save warning.

I think the issue is that the IM-based loaders/savers haven't been reworked for the source/target API, see:
#2051

For loading, there's a fallback mechanism that uses the old file/buffer-based loaders. However, the im.WriteToStream() and im.WriteToTarget() functions lack such a fallback.

As a result, this won't work:

using var input = File.OpenRead("MARBLES.bmp");

using var image = Image.NewFromStream(input, access: Enums.Access.Sequential);
Console.WriteLine(image.ToString());

using var output = File.OpenWrite("x.bmp");
image.WriteToStream(output, ".bmp");

Instead, you need to use:

@@ -3,5 +3,4 @@ using var input = File.OpenRead("MARBLES.bmp");
 using var image = Image.NewFromStream(input, access: Enums.Access.Sequential);
 Console.WriteLine(image.ToString());
 
-using var output = File.OpenWrite("x.bmp");
-image.WriteToStream(output, ".bmp");
+image.WriteToFile("x.bmp");

L3tum Jul 10, 2024
Author

I've already got a lot of code to specifically deal with BMP :( So the difference in API needed is unfortunately already handled.

I actually need to apologize. It turns out that my Debian distribution was correctly pulling the latest libvips build, while my non-Debian/scratch distribution used a cached version from 6 months ago, which didn't contain ImageMagick yet. Forcing a docker pull correctly uses the latest version and everything works...

Thank you very much for your help! Your script has been invaluable in getting this up and running

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BMP does not work in very specific situations ("bmp is not a known buffer format") #4035

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

BMP does not work in very specific situations ("bmp is not a known buffer format") #4035

Uh oh!

L3tum Jul 9, 2024

Replies: 1 comment · 3 replies

Uh oh!

kleisauke Jul 9, 2024 Maintainer

Uh oh!

L3tum Jul 10, 2024 Author

Uh oh!

kleisauke Jul 10, 2024 Maintainer

Uh oh!

L3tum Jul 10, 2024 Author

L3tum
Jul 9, 2024

Replies: 1 comment 3 replies

kleisauke
Jul 9, 2024
Maintainer

L3tum Jul 10, 2024
Author

kleisauke Jul 10, 2024
Maintainer

L3tum Jul 10, 2024
Author