gh-132257: Remove -flto-partition=none for Linux LTO builds #132258

Fidget-Spinner · 2025-04-08T09:09:26Z

Change the default LTO flags on GCC to not pass -flto-partition=none, and allow parallelization of LTO. This has a multiple factor speedup for LTO build times on GCC, with no noticeable loss in performance.

On newer make and newer GCC, this passes the jobserver automatically to GCC (or more like GCC grabs it from the env vars).

On older make, this will have benign warnings about serial compilation. It's safe to ignore them.

Issue: Our GCC LTO flags can be improved #132257

corona10 · 2025-04-08T10:50:43Z

I just want to check that it’s the compatible with old gcc too?

Fidget-Spinner · 2025-04-08T11:25:19Z

How old are we talking about? I dont know the performance on ancient gcc, but I tested on the faster cpython machines which purportedly uses gcc 9 and 11, and it had no significant slowdown. GCC 9 was released in 2019 (6 years old by now).

Ancient gcc night have some perf loss, but if thats the case we should not backport this so it releases with 3.14 and distros that use it will be new and bundle new gcc.

Fidget-Spinner · 2025-04-08T11:28:48Z

Ah apparently flyo=auto is only gcc 10 and above. Good catch! I will try to guard this on gcc version

Fidget-Spinner · 2025-04-08T11:30:34Z

Alternatively i can just remove the auto since that is the default behaviiot apparently https://www.phoronix.com/news/GCC-10-LTO-flto-Available-Cores

corona10 · 2025-04-08T11:56:15Z

IIRC we should consider gcc that supports C11.

Yhg1s · 2025-04-08T13:17:00Z

Can you please include the news item's text in the PR description/commit message? That way it's much easier to see what a change meant to do when looking at commit messages.

Fidget-Spinner · 2025-04-08T14:08:30Z

IIRC we should consider gcc that supports C11.

First version of GCC to support C11 is 4-5ish. First version that seems to work with -flto=auto flag is 8.1. That was released in 2018. I will just guard to not use this flag on anything that is GCC 7 or earlier.

Fidget-Spinner · 2025-04-08T14:15:05Z

@corona10 ok I pushed a commit to support older compilers by not turning on the flag if it detects GCC 7 or below.

corona10 · 2025-04-09T16:51:44Z

I will take a look at ths PR by today :) Sorry for the delay.

configure.ac

AA-Turner · 2025-04-09T21:47:00Z

Buildbot compilation times for eedba32 (#132258):

	Fedora LTO (aarch)	Fedora LTO (AMD64)
This PR `eedba32` (#132258)	1m45s	1m56s
This PR `09205a0` (#132258)	1m35s	2m01s

Prev run 1	9m23s	6m01s
Prev run 2	8m58s	6m11s
Prev run 3	9m59s	6m28s
Prev run 4	10m10s	6m38s

Clearly a sample size of N=1, but encouraging results!

A

eli-schwartz

Thank you. Overall, I'm quite happy to see -flto-partition dropped. I do have a small nit though.

configure.ac

Fidget-Spinner · 2025-04-10T11:42:47Z

@corona10 note that this might expose more concurrency problems in our makefile. As apparently the previous makefile did not respect jobserver, so was not parallelising stuff properly!

configure.ac

Makefile.pre.in

This reverts commit c7f4cc4.

Fidget-Spinner · 2025-04-10T16:38:13Z

@eli-schwartz or @thesamesam can you please check out if this speeds up CPython LTO build times on Gentoo with newer GNU make? I tried building make from source on Ubuntu 22.04 but apparently the autoconf on there is 2.71 which is too old to build GNU make from source. https://launchpad.net/ubuntu/+source/autoconf

thesamesam · 2025-04-10T17:26:55Z

Sure, will try, but FWIW, the GNU Make release tarballs should already be autoreconf'd, so only need to run ./configure in those.

Fidget-Spinner · 2025-04-10T17:39:15Z

A thanks Sam I pulled from source not the tarball. Will try again.

Fidget-Spinner · 2025-04-10T17:49:56Z

Ok can confirm this works wonderfully with GNU make 4.4.1. Everything is auto-parallelised. Yipee! Thanks Gentoo maintainers :)

eli-schwartz · 2025-04-10T18:00:40Z

@eli-schwartz or @thesamesam can you please check out if this speeds up CPython LTO build times on Gentoo with newer GNU make?

Make-only, with the patch, -j8:

2m22.199s

Without the patch:

4m32.842s

I tried building make from source on Ubuntu 22.04 but apparently the autoconf on there is 2.71 which is too old to build GNU make from source. https://launchpad.net/ubuntu/+source/autoconf

GNU Make from source only requires autoconf 2.69 anyway. (Notwithstanding what Sam said about the release tarball).

bedevere-bot · 2025-04-11T08:45:22Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Fedora Stable LTO 3.x (tier-2) has failed when building commit bc0b94b.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/336/builds/6654) and take a look at the build logs.
Check if the failure is related to this commit (bc0b94b) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/336/builds/6654

Failed tests:

test_perf_profiler

Failed subtests:

test_python_calls_appear_in_the_stack_if_perf_activated - test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/Lib/threading.py", line 1079, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/Lib/threading.py", line 1021, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/Lib/test/test_interpreters/test_stress.py", line 30, in task
    interp = interpreters.create()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/Lib/test/support/interpreters/__init__.py", line 76, in create
    id = _interpreters.create(reqrefs=True)
interpreters.InterpreterError: interpreter creation failed
k


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/Lib/test/test_perf_profiler.py", line 364, in test_python_calls_appear_in_the_stack_if_perf_activated
    self.assertIn(f"py::foo:{script}", stdout)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'py::foo:/tmp/test_python_shw4qp61/tmpxvu1t7hq/perftest.py' not found in 'python  778380 1516589.097708:          1 cycles:Pu: \n\t    ffffb44ffac0 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)\n\npython  778380 1516589.097743:          1 cycles:Pu: \n\tffffaa282d37fc78 [unknown] ([unknown])\n\tffffaa282d38049c [unknown] ([unknown])\n\tffffaa282bf215e4 [unknown] ([unknown])\n\t    ffffb44ffac0 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)\n\npython  778380 1516589.098060:          1 cycles:Pu: \n\t    ffffb4501600 mmap64+0x0 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44ee75b _dl_sysdep_read_whole_file+0x9b (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44f6a0f _dl_load_cache_lookup+0x127 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44ec2c3 _dl_map_object+0x377 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44e75bf openaux+0x3f (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44e6303 _dl_catch_exception+0x63 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44e7b33 _dl_map_object_deps+0x553 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44fd19f dl_main+0x139f (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44fa5ff _dl_sysdep_start+0x1df (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44fbb17 _dl_start_final+0x5ab (inlined)\n\t    ffffb44fbb17 _dl_start+0x5ab (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44ffad3 _start+0x13 (/usr/lib/ld-linux-aarch64.so.1)\n\npython  778380 1516589.098108:        321 cycles:Pu: \n\t    ffffb4501180 __GI___close_nocancel+0x0 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb44ee72b _dl_sysdep_read_whole_file+0x6b (/usr/lib/ld-linux-aa
re_create_interpreter+0xb23 (inlined)\n\t          7282cf pyinit_config+0xb23 (inlined)\n\t          7282cf pyinit_core.constprop.0+0xb23 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          64c01f Py_InitializeFromConfig+0x37 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          6855c7 pymain_init+0x13f (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          686d2f pymain_main+0x27 (inlined)\n\t          686d2f Py_BytesMain+0x27 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t    ffffb429625b __libc_start_call_main+0x7b (/usr/lib64/libc.so.6)\n\t    ffffb429633b __libc_start_main@@GLIBC_2.34+0x9b (/usr/lib64/libc.so.6)\n\t          41f5af _start+0x2f (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\npython  778380 1516589.101192:     826833 cycles:Pu: \n\t    ffffb431ac58 __strlen_asimd+0x58 (/usr/lib64/libc.so.6)\n\t          545773 PyUnicode_FromString+0x13 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          48f0fb PyUnicode_InternFromString+0x5b (inlined)\n\t          48f0fb descr_new.lto_priv.0+0x5b (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          5237f3 PyDescr_NewMethod+0x1b3 (inlined)\n\t          5237f3 type_add_method.lto_priv.0+0x1b3 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          5259af type_add_methods+0x24f (inlined)\n\t          5259af type_ready_fill_dict+0x24f (inlined)\n\t          5259af type_ready.lto_priv.0+0x24f (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          6452a7 init_static_type+0xa7 (inlined)\n\t          6452a7 _PyStaticType_InitBuiltin+0xa7 (inlined)\n\t          6452a7 _PyTypes_InitTypes+0xa7 (inlined)\n\t          6452a7 pycore_init_types+0xa7 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          646843 pycore_interp_init.lto_priv.0+0x103 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          7282df pyinit_config+0xb33 (inlined)\n\t          7282df pyinit_core.constprop.0+0xb33 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          64c01f Py_InitializeFromConfig+0x37 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          6855c7 pymain_init+0x13f (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          686d2f pymain_main+0x27 (inlined)\n\t          686d2f Py_BytesMain+0x27 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t    ffffb429625b __libc_start_call_main+0x7b (/usr/lib64/libc.so.6)\n\t    ffffb429633b __libc_start_main@@GLIBC_2.34+0x9b (/usr/lib64/libc.so.6)\n\t          41f5af _start+0x2f (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\npython  778380 1516589.102775:    2460759 cycles:Pu: \n\t          51f890 PyTuple_SET_ITEM+0x420 (inlined)\n\t          51f890 mro_implementation_unlocked.lto_priv.0+0x420 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          73d343 mro_invoke+0x323 (inlined)\n\t          73d343 mro_internal_unlocked.isra.0+0x323 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          5257df type_ready_mro+0x7f (inlined)\n\t          5257df type_ready.lto_priv.0+0x7f (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          52ad1b init_static_type+0xaf (inlined)\n\t          52ad1b _PyStaticType_InitBuiltin+0xaf (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          517817 _PyStructSequence_InitBuiltinWithFlags+0x1b7 (/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto/build/python)\n\t          670aab _PyStructSequence_InitBuiltin+0x5eb (inlined)\n\t          670aab _PySys_InitCore+0x5eb (inlined)\n\t          670aab _PySys_

Remove -flto-partition=none for Linux LTO builds

8891cd2

Fidget-Spinner requested review from erlend-aasland and corona10 as code owners April 8, 2025 09:09

bedevere-app bot added the awaiting core review label Apr 8, 2025

bedevere-app bot mentioned this pull request Apr 8, 2025

Our GCC LTO flags can be improved #132257

Closed

📜🤖 Added by blurb_it.

23568cd

Support older compilers

09205a0

gpshead approved these changes Apr 9, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Apr 9, 2025

corona10 self-assigned this Apr 9, 2025

AA-Turner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 9, 2025

This comment was marked as resolved.

Sign in to view

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 9, 2025

corona10 reviewed Apr 9, 2025

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

Address review

eedba32

AA-Turner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 9, 2025

This comment was marked as resolved.

Sign in to view

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 9, 2025

eli-schwartz approved these changes Apr 10, 2025

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

Fidget-Spinner added 2 commits April 10, 2025 19:38

Pass jobserver env to all commands

c7f4cc4

autoconf regen

9e013cf

This comment was marked as resolved.

Sign in to view

Yhg1s reviewed Apr 10, 2025

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

Makefile.pre.in Outdated Show resolved Hide resolved

Fidget-Spinner added 2 commits April 10, 2025 20:25

Revert "Pass jobserver env to all commands"

6257b0e

This reverts commit c7f4cc4.

Revert most of the + to the bare minimum

e438b4c

This comment was marked as resolved.

Sign in to view

Fidget-Spinner added 2 commits April 10, 2025 23:23

Remove the +

61c8e33

Update 2025-04-08-09-11-32.gh-issue-132257.oZWBV-.rst

3628388

Yhg1s approved these changes Apr 10, 2025

View reviewed changes

Fidget-Spinner merged commit bc0b94b into python:main Apr 11, 2025
38 checks passed

bedevere-app bot removed the awaiting merge label Apr 11, 2025

Fidget-Spinner deleted the lto-fix branch April 11, 2025 08:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-132257: Remove -flto-partition=none for Linux LTO builds #132258

gh-132257: Remove -flto-partition=none for Linux LTO builds #132258

Fidget-Spinner commented Apr 8, 2025 •

edited

Loading

corona10 commented Apr 8, 2025

Fidget-Spinner commented Apr 8, 2025 •

edited

Loading

Fidget-Spinner commented Apr 8, 2025

Fidget-Spinner commented Apr 8, 2025

corona10 commented Apr 8, 2025

Yhg1s commented Apr 8, 2025

Fidget-Spinner commented Apr 8, 2025

Fidget-Spinner commented Apr 8, 2025

corona10 commented Apr 9, 2025

This comment was marked as resolved.

AA-Turner commented Apr 9, 2025 •

edited

Loading

This comment was marked as resolved.

eli-schwartz left a comment

Fidget-Spinner commented Apr 10, 2025

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Fidget-Spinner commented Apr 10, 2025

thesamesam commented Apr 10, 2025

Fidget-Spinner commented Apr 10, 2025

Fidget-Spinner commented Apr 10, 2025

eli-schwartz commented Apr 10, 2025

bedevere-bot commented Apr 11, 2025

gh-132257: Remove -flto-partition=none for Linux LTO builds #132258

gh-132257: Remove -flto-partition=none for Linux LTO builds #132258

Conversation

Fidget-Spinner commented Apr 8, 2025 • edited Loading

corona10 commented Apr 8, 2025

Fidget-Spinner commented Apr 8, 2025 • edited Loading

Fidget-Spinner commented Apr 8, 2025

Fidget-Spinner commented Apr 8, 2025

corona10 commented Apr 8, 2025

Yhg1s commented Apr 8, 2025

Fidget-Spinner commented Apr 8, 2025

Fidget-Spinner commented Apr 8, 2025

corona10 commented Apr 9, 2025

This comment was marked as resolved.

AA-Turner commented Apr 9, 2025 • edited Loading

This comment was marked as resolved.

eli-schwartz left a comment

Choose a reason for hiding this comment

Fidget-Spinner commented Apr 10, 2025

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Fidget-Spinner commented Apr 10, 2025

thesamesam commented Apr 10, 2025

Fidget-Spinner commented Apr 10, 2025

Fidget-Spinner commented Apr 10, 2025

eli-schwartz commented Apr 10, 2025

bedevere-bot commented Apr 11, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Fidget-Spinner commented Apr 8, 2025 •

edited

Loading

Fidget-Spinner commented Apr 8, 2025 •

edited

Loading

AA-Turner commented Apr 9, 2025 •

edited

Loading