CI: revert LDFLAGS sysroot work-around#22535

lesteve wants to merge 4 commits intoscikit-learn:mainscikit-learn/scikit-learn:mainfrom

lesteve:remove-build-against-system-fixlesteve/scikit-learn:remove-build-against-system-fixCopy head branch name to clipboard