Skip to content

PGO-only builds on Linux/macOS for embedded use #470

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Edward-Knight opened this issue Jan 3, 2025 · 5 comments
Closed

PGO-only builds on Linux/macOS for embedded use #470

Edward-Knight opened this issue Jan 3, 2025 · 5 comments
Labels
variants Modifying the published variants of distributions

Comments

@Edward-Knight
Copy link
Contributor

Edward-Knight commented Jan 3, 2025

#442 removed PGO-only builds for Linux/macOS, as PGO+LTO builds were available.

We use the PBS builds both as a standalone application and as an embedded Python runtime. We were using the PGO-only builds as we encountered difficulties linking to the PGO+LTO builds. I'm not particularly well versed on the subject, but the report from a coworker was:

The LTO packages ship a LLVM bitcode libpython.a rather than fully compiled library and this means that we cannot use the library in <our application> without also matching the compiler (which we don't do as we use GCC on most platforms).

Would you be interested in re-enabling the PGO-only builds? Is there another way around our link problem, other than matching compilers?

@indygreg
Copy link
Collaborator

indygreg commented Jan 3, 2025

Your coworker is correct about the LTO object files being LLVM bitcode.

To link them reliably you need to use the exact same LLVM toolchain that created them.

Path of least resistance is to avoid LTO.

@zanieb
Copy link
Member

zanieb commented Jan 3, 2025

We can consider it for sure, it's just a lot more artifacts to build if we include PGO-only builds.

@zanieb
Copy link
Member

zanieb commented Jan 5, 2025

Is it feasible to build these yourself? Or do you need to target a bunch of platforms?

@zanieb zanieb added the variants Modifying the published variants of distributions label Jan 5, 2025
@Edward-Knight
Copy link
Contributor Author

Edward-Knight commented Jan 6, 2025

We are currently using the following builds:

  • macOS
    • AArch64 (aarch64-apple-darwin, pgo-full)
    • x86-64 (x86_64-apple-darwin, pgo-full)
  • Linux
    • AArch64 (aarch64-unknown-linux-gnu, noopt-full) (no PGO available)
    • x86-64 (x86_64_v2-unknown-linux-gnu, pgo-full)
  • Non-PBS builds

We technically only need to embed the Python interpreter on iOS (due to only being allowed one process per app),
but for easier testing and consistency between builds we also do this on macOS and Linux.

My preference is to maintain as little as possible myself and to reduce deviation between our platforms. Assuming that PGO remains a valid build configuration, it would be possible for us to fork and build ourselves. I imagine this will be our fallback position (I don't love the idea of using the debug-full builds).

As an aside we use Conan to build our other C/C++ dependencies internally. We haven't explored using it for CPython, but that might also be a way forward for us (I can see a recipe does already exist with some patches)

@Edward-Knight
Copy link
Contributor Author

We have dropped our use of embedded Python in macOS and Linux (and iOS), and so no longer need this.

As an aside, we did experiment with using Conan to build CPython. We found the recipe worked fine, but there do seem to be some quirks for macOS. After compiling with our initial attempt we were missing the standard library, but haven't looked into it further.

Will close the issue since I don't see anyone else asking for this variant. Should this change in the future I guess this issue could be reopened or a new issue created 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
variants Modifying the published variants of distributions
Projects
None yet
Development

No branches or pull requests

3 participants