Skip to content

plugins: kafka: fix cmake cross compile error #9600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ThomasDevoogdt
Copy link
Contributor

KAFKA_INCLUDEDIR is not set if FLB_PREFER_SYSTEM_LIB_KAFKA is not used, when cross-compiling, it just translates to -I/librdkafka, which is not allowed. Fix this by only including KAFKA_INCLUDEDIR if really set.

x86_64-linux-gcc: ERROR: unsafe header/library path used in cross-compilation: '-I/librdkafka'


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • [N/A] Example configuration file for the change
  • [N/A] Debug log output from testing the change
  • [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@ThomasDevoogdt ThomasDevoogdt changed the title fix kafka cross compile error kafka: fix cross compile error Nov 16, 2024
@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from b398800 to c96d616 Compare November 17, 2024 08:01
@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from c96d616 to abc06b7 Compare December 15, 2024 14:16
@ThomasDevoogdt
Copy link
Contributor Author

@edsiper It has been a while. Any chance that this gets approved? I try to reduce my own patch list.

@ThomasDevoogdt
Copy link
Contributor Author

@edsiper Happy New Year! Can you have a look at this PR please?

@ThomasDevoogdt
Copy link
Contributor Author

@edsiper @leonardo-albertovich @fujimotos @koleini This PR is open for a while. Is it possible to have a look?

@ThomasDevoogdt ThomasDevoogdt changed the title kafka: fix cross compile error lib: kafka: fix cross compile error Jan 10, 2025
@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from abc06b7 to 5c51513 Compare January 10, 2025 07:14
@patrick-stephens patrick-stephens added the ok-package-test Run PR packaging tests label Jan 13, 2025
@ThomasDevoogdt
Copy link
Contributor Author

@patrick-stephens
Copy link
Contributor

Can we rebase to see if it resolves the unit test failures?

@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from 5c51513 to 18017d7 Compare March 26, 2025 14:08
@ThomasDevoogdt
Copy link
Contributor Author

Can we rebase to see if it resolves the unit test failures?

I rebased it.

@ThomasDevoogdt
Copy link
Contributor Author

Can we rebase to see if it resolves the unit test failures?

@patrick-stephens rebased it, and all checks are passing.

@ThomasDevoogdt
Copy link
Contributor Author

@patrick-stephens what is the next step to get this merged? I also have other open PRs which are approved, but not merged yet. Can you check my other PRs also? https://github.com/fluent/fluent-bit/pulls/ThomasDevoogdt

@patrick-stephens
Copy link
Contributor

@edsiper will be merging when he is happy to put it in a release - be aware we're finalising the 4.0 release now.

@ThomasDevoogdt
Copy link
Contributor Author

@edsiper will be merging when he is happy to put it in a release - be aware we're finalising the 4.0 release now.

@patrick-stephens @edsiper I see that 4.0.0 has landed, this is the ideal moment to consider this PR.

@ThomasDevoogdt
Copy link
Contributor Author

@patrick-stephens can you also check this one, while at it?

@patrick-stephens
Copy link
Contributor

Looks fine to me, just want to check @edsiper is happy with the commit title

@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from 18017d7 to 86fb988 Compare May 7, 2025 07:33
@ThomasDevoogdt
Copy link
Contributor Author

Looks fine to me, just want to check @edsiper is happy with the commit title

@patrick-stephens @edsiper I couldn't really find a similar example for the git message log. But I changed it a bit to what is probably more right. Can you have a look at this PR?

@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from 86fb988 to 8c60cbf Compare May 8, 2025 20:48
@ThomasDevoogdt ThomasDevoogdt changed the title lib: kafka: fix cross compile error plugins: kafka: fix cmake cross compile error May 8, 2025
@ThomasDevoogdt
Copy link
Contributor Author

@cosmo0920 This is another cross compilation issue I encountered.

@cosmo0920
Copy link
Contributor

@cosmo0920 This is another cross compilation issue I encountered.

How could we check this issue? Just using gcc-aarch64-linux-gnu on x86_64 platform to cross compiling is enough?

@ThomasDevoogdt
Copy link
Contributor Author

@cosmo0920 This is another cross compilation issue I encountered.

How could we check this issue? Just using gcc-aarch64-linux-gnu on x86_64 platform to cross compiling is enough?

It took me some time to reproduce, but these are the steps:

git clone https://github.com/buildroot/buildroot.git
cd buildroot

rm package/fluent-bit/0001-plugins-kafka-fix-cmake-cross-compile-error.patch
echo support/config-fragments/autobuild/bootlin-x86-64-glibc.config,x86_64 > toolchain-configs.csv

./utils/test-pkg -p fluent-bit -t toolchain-configs.csv

And follow the output in another shell:

tail -F br-test-pkg/bootlin-x86-64-glibc/logfile

You can also fix stuff by going to the work folder, and change the current checkout:

cd br-test-pkg/bootlin-x86-64-glibc/
# edit build/fluent-bit-4.0.2/
make

You will see this error:

[335/1140] Building C object plugins/custom_calyptia/CMakeFiles/flb-plugin-custom_calyptia.dir/calyptia.c.o
FAILED: plugins/custom_calyptia/CMakeFiles/flb-plugin-custom_calyptia.dir/calyptia.c.o 
/home/thomas/br-test-pkg/bootlin-x86-64-glibc/host/bin/x86_64-linux-gcc --sysroot=/home/thomas/br-test-pkg/bootlin-x86-64-glibc/host/x86_64-buildroot-linux-gnu/sysroot -DFLB_EVENT_LOOP_AUTO_DISCOVERY -DFLB_HAVE_ACCEPT4 -DFLB_HAVE_ATTRIBUTE_ALLOC_SIZE -DFLB_HAVE_AWS -DFLB_HAVE_AWS_CREDENTIAL_PROCESS -DFLB_HAVE_FORK -DFLB_HAVE_GETENTROPY -DFLB_HAVE_GETENTROPY_SYS_RANDOM -DFLB_HAVE_GMTOFF -DFLB_HAVE_HTTP_SERVER -DFLB_HAVE_INOTIFY -DFLB_HAVE_IN_STORAGE_BACKLOG -DFLB_HAVE_LIBYAML -DFLB_HAVE_LITTLE_ENDIAN_SYSTEM -DFLB_HAVE_METRICS -DFLB_HAVE_OPENSSL -DFLB_HAVE_PARSER -DFLB_HAVE_PROFILES -DFLB_HAVE_PROXY_GO -DFLB_HAVE_RECORD_ACCESSOR -DFLB_HAVE_REGEX -DFLB_HAVE_SIGNV4 -DFLB_HAVE_SQLDB -DFLB_HAVE_STREAM_PROCESSOR -DFLB_HAVE_SYS_WAIT_H -DFLB_HAVE_TLS -DFLB_HAVE_UNICODE_ENCODER -DFLB_HAVE_UNIX_SOCKET -DFLB_HAVE_UTF8_ENCODER -DFLB_MSGPACK_TO_JSON_INIT_BUFFER_SIZE=2.0 -DFLB_MSGPACK_TO_JSON_REALLOC_BUFFER_SIZE=0.10 -DFLB_SYSTEM_LINUX -DJSMN_PARENT_LINKS -DJSMN_STRICT -DLIBRDKAFKA_STATICLIB -DMINIZ_STATIC_DEFINE -DMPACK_EXTENSIONS=1 -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/fluent-otel-proto/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/fluent-otel-proto/proto_c -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/cfl/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/cfl/lib/xxhash -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/flb_libco -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/rbtree -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/chunkio/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/monkey/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/monkey/include/monkey -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/mpack-amalgamation-1.1.1/src -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/miniz -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/onigmo -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/snappy-fef67ac -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/cmetrics/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/ctraces/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/cprofiles/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/lwrb/lwrb/src/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/jansson-e23f558/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/cmetrics -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/tutf8e/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/msgpack-c/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/sqlite-amalgamation-3450200 -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/librdkafka-2.8.0/src -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/cfl/lib/xxhash/cmake_unofficial/.. -I/librdkafka -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/lib/librdkafka-2.8.0/generated/dummy -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/plugins/filter_geoip2/libmaxminddb-1.12.2 -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/plugins/filter_geoip2/libmaxminddb-1.12.2/include -I/home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/plugins/filter_geoip2/libmaxminddb-1.12.2/generated -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g0 -D_FORTIFY_SOURCE=1 -Wall -D__FLB_FILENAME__=__FILE__ -O2 -g -DNDEBUG -fPIC -MD -MT plugins/custom_calyptia/CMakeFiles/flb-plugin-custom_calyptia.dir/calyptia.c.o -MF plugins/custom_calyptia/CMakeFiles/flb-plugin-custom_calyptia.dir/calyptia.c.o.d -o plugins/custom_calyptia/CMakeFiles/flb-plugin-custom_calyptia.dir/calyptia.c.o -c /home/thomas/br-test-pkg/bootlin-x86-64-glibc/build/fluent-bit-4.0.2/plugins/custom_calyptia/calyptia.c
x86_64-linux-gcc: ERROR: unsafe header/library path used in cross-compilation: '-I/librdkafka'

Copy link
Contributor

@cosmo0920 cosmo0920 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the behavior around this line: https://github.com/fluent/fluent-bit/blob/master/CMakeLists.txt#L1144
Yep, KAFKA_INCLUDEDIR is originated from there and this change is valid. 👍

@cosmo0920 cosmo0920 added this to the Fluent Bit v4.0.4 milestone Jun 5, 2025
@cosmo0920
Copy link
Contributor

cosmo0920 commented Jun 5, 2025

Note: This patch is originated from Fluent Bit side of glitches among CMakeLists.txt and kafka plugins. So, we need to handle this compilation error on this repo.

@ThomasDevoogdt
Copy link
Contributor Author

@cosmo0920 It has been approved for a while, can this be merged?

@edsiper
Copy link
Member

edsiper commented Jul 9, 2025

let's resolve the conflicts since we had recent changes in the Kafka layer and make it part of the next release

@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from 8c60cbf to 9d51c13 Compare July 9, 2025 05:49
@ThomasDevoogdt
Copy link
Contributor Author

let's resolve the conflicts since we had recent changes in the Kafka layer and make it part of the next release

@edsiper I rebased the branch. Perhaps you can still take it along?

@patrick-stephens
Copy link
Contributor

Looks like CentOS stream dependencies have gone even more incompatible...

@ThomasDevoogdt
Copy link
Contributor Author

Looks like CentOS stream dependencies have gone even more incompatible...

I don't think it is related with this PR, can you restore the milestone label?

@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from 9d51c13 to fb46e10 Compare July 14, 2025 21:00
@ThomasDevoogdt
Copy link
Contributor Author

@edsiper @patrick-stephens can you have a look, this has been approved before, I don't see any issues anymore, can this be merged?

@patrick-stephens
Copy link
Contributor

Yeah I am fine with it, I think the Debian failures are unrelated.

KAFKA_INCLUDEDIR is not set if FLB_PREFER_SYSTEM_LIB_KAFKA is not used,
when cross-compiling, it just translates to -I/librdkafka, which is not allowed.
Fix this by only including KAFKA_INCLUDEDIR if really set.

x86_64-linux-gcc: ERROR: unsafe header/library path used in cross-compilation: '-I/librdkafka'

Signed-off-by: Thomas Devoogdt <thomas@devoogdt.com>
@ThomasDevoogdt ThomasDevoogdt force-pushed the bugfix/cross-compile-kafka branch from fb46e10 to dcecf8f Compare July 19, 2025 13:35
@ThomasDevoogdt
Copy link
Contributor Author

@patrick-stephens @edsiper It's again a bit silent around this PR. Everything is approved and ready to be merged, on what do we exactly wait? Can you just merge this one?

@ThomasDevoogdt
Copy link
Contributor Author

@cosmo0920 Can you put this PR back on another more milestone, I have the feeling that next means forgotten. Or just merge this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-required ok-package-test Run PR packaging tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants