Skip to content

GH-132983: Support finding libzstd without pkg-config #133550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

emmatyping
Copy link
Member

@emmatyping emmatyping commented May 7, 2025

GH-133479 removed the logic to check for libzstd outside of pkg-config. This commit adds that logic back with a check for the version so that users can provide their own libzstd. This is to bring parity with lzma, bz2, and zlib detection.

The issues previously seen on RHEL 8 were due to how the checks are ordered. After #133479, if a library version was too old, the PKG_CHECK_MODULES macro would take the "not found" path. This would then search for and find zstd.h and a libzstd with the symbol ZDICT_finalizeDictionary, but then compilation would fail when the header check occured. #133502 removed this path solving the issue on RHEL 8. However, this also make it impossible for a user to set a custom build of libzstd as the dependency (as they can do for liblzma, libbz2, etc.).

In this PR we add this path back, but add an additional check for the version of the libraries. If a user/third party packager would like to point to their own libzstd, they can do so. We avoid the issue #133479 was meaning to solve by ensuring that the version check only runs if we already have the zstd.h header and libzstd library found.

pythonGH-133479 removed the logic to check for libzstd outside of pkg-config.
This commit adds that logic back with a check for the version so that
users can provide their own libzstd. This is to bring parity with lzma,
bz2, and zlib detection.
@emmatyping emmatyping requested review from gpshead and AA-Turner May 7, 2025 03:07
@emmatyping emmatyping changed the title Support finding libzstd not in pkg-config GH-132983: Support finding libzstd not in pkg-config May 7, 2025
@emmatyping emmatyping changed the title GH-132983: Support finding libzstd not in pkg-config GH-132983: Support finding libzstd without pkg-config May 7, 2025
@emmatyping
Copy link
Member Author

!buildbot RHEL8

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @emmatyping for commit f3dd7f7 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133550%2Fmerge

The command will test the builders whose names match following regular expression: RHEL8

The builders matched are:

  • PPC64LE RHEL8 LTO PR
  • PPC64LE RHEL8 Refleaks PR
  • AMD64 RHEL8 LTO + PGO PR
  • aarch64 RHEL8 PR
  • aarch64 RHEL8 Refleaks PR
  • AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
  • AMD64 RHEL8 PR
  • aarch64 RHEL8 LTO + PGO PR
  • aarch64 RHEL8 LTO PR
  • s390x RHEL8 Refleaks PR
  • PPC64LE RHEL8 LTO + PGO PR
  • s390x RHEL8 PR
  • s390x RHEL8 LTO PR
  • s390x RHEL8 LTO + PGO PR
  • AMD64 RHEL8 FIPS No Builtin Hashes PR
  • AMD64 RHEL8 Refleaks PR
  • AMD64 RHEL8 LTO PR
  • PPC64LE RHEL8 PR

@emmatyping
Copy link
Member Author

Scheduled RHEL 8 but I also want to double check the ghcr.io/cirruslabs/macos-runner:sonoma job.

@emmatyping
Copy link
Member Author

emmatyping commented May 7, 2025

Okay this is looking good.
RHEL 8:

...
checking for libzstd >= 1.4.5... no
checking for zstd.h... no
checking for zdict.h... no
...
checking for stdlib extension module _zstd... missing
...

macOS Sonoma:

...
checking for libzstd >= 1.4.5... yes
...
checking for stdlib extension module _zstd... yes
...

The AMD64 RHEL8 FIPS No Builtin Hashes PR buildbot failure looks unrelated.

@AA-Turner
Copy link
Member

I've opened #133565 as an alternative with slightly different (and I think also faster) autoconf syntax, what do you think?

A

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
@emmatyping
Copy link
Member Author

!buildbot RHEL8

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @emmatyping for commit 235fe69 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133550%2Fmerge

The command will test the builders whose names match following regular expression: RHEL8

The builders matched are:

  • PPC64LE RHEL8 LTO PR
  • PPC64LE RHEL8 Refleaks PR
  • AMD64 RHEL8 LTO + PGO PR
  • aarch64 RHEL8 PR
  • aarch64 RHEL8 Refleaks PR
  • AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
  • AMD64 RHEL8 PR
  • aarch64 RHEL8 LTO + PGO PR
  • aarch64 RHEL8 LTO PR
  • s390x RHEL8 Refleaks PR
  • PPC64LE RHEL8 LTO + PGO PR
  • s390x RHEL8 PR
  • s390x RHEL8 LTO PR
  • s390x RHEL8 LTO + PGO PR
  • AMD64 RHEL8 FIPS No Builtin Hashes PR
  • AMD64 RHEL8 Refleaks PR
  • AMD64 RHEL8 LTO PR
  • PPC64LE RHEL8 PR

Copy link
Contributor

@erlend-aasland erlend-aasland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this seems correct to me! It would be unfortunate if we had a different set of configure rules for libzstd than for other third-party dependencies.

@emmatyping
Copy link
Member Author

Closing in favor of #133565

@emmatyping emmatyping closed this May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants