Skip to content

Work around a broken shared detection on some platforms #300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Dec 7, 2016

Conversation

den-run-ai
Copy link
Contributor

@mikofski
Copy link

mikofski commented Dec 4, 2016

This seems like an issue with xamarin mono project, not .NET

Seems like a shame to have it always turned off, for a small percentage of PythonNET users

Why not do some sort of check for Linux or darwin, then disable, but if win32 use flag?

@den-run-ai
Copy link
Contributor Author

den-run-ai commented Dec 4, 2016

@mikofski can you please explain how Mono is to blame in more details, especially if you can point to specific code?

@den-run-ai
Copy link
Contributor Author

@tonyroberts @filmor @vmuriart please review. Note that I marked this pull request as release-blocking.

@tonyroberts
Copy link
Contributor

This change may fix a problem for some cases, but it will break it for others.

If that flag isn't reliable because of a bug in some distributions, why not check another way instead of ignoring it? Checking the output of ldd on the Python executable would tell you if it's been linked dynamically or not.

@den-run-ai
Copy link
Contributor Author

@tonyroberts @filmor this was a bit of learning for me. See analysis below.

On Linux Mint it shows that python 2/3 are dynamically linked:

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.sysconfig import get_config_var
>>> get_config_var("Py_ENABLE_SHARED")
1

Python 3.4.3 (default, Sep 14 2016, 12:36:27) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.sysconfig import get_config_var
>>> get_config_var("Py_ENABLE_SHARED")
1

here is the output of ldd on Linux Mint with python 2/3:

dta@dta-Inspiron-N5050 ~ $ ldd `which python`
	linux-vdso.so.1 =>  (0x00007fff7e9ec000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f60536ed000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6053328000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6053123000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f6052f20000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6052d07000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6052a00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f605392e000)
dta@dta-Inspiron-N5050 ~ $ ldd `which python3`
	linux-vdso.so.1 =>  (0x00007fff1a6e3000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd057cb8000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd0578f3000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd0576ee000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fd0574eb000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fd0572c1000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd0570a7000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd056da1000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd057ef9000)

According to this wiki this most likely means that python2/python3 were build dynamically:

https://wiki.python.org/moin/BuildStatically

I tried ldd with statically build python from source and indeed it shows the correct message:

cd ./Python-2.7.12/
./configure --disable-shared LDFLAGS="-static -static-libgcc" CPPFLAGS="-static"
make LDFLAGS="-static" LINKFORSHARED=" "
ldd ./python 

with output:

not a dynamic executable

The problem is that even when python is dynamically linked, import clr fails! Most likely this is because the dynamic resolution is different from the expected. But forcing PYTHON_WITHOUT_ENABLE_SHARED works great. Supposedly there is a fix with setting LD_LIBRARY_PATH, but this changes Linux settings system-wide.

@tonyroberts so can you give examples on which Linux systems and how this would still fail?

@tonyroberts
Copy link
Contributor

@denfromufa Some versions of the Python executable are built with the python runtime embedded in them, and others are built with it as an external shared object (.so) file. For the ones that use a separate .so file, extensions like clr must also be built to use the same shared object. Ones that embed that code into the python executable must have extension modules built slightly differently (hence the Py_ENABLED_SHARED option, which should be set if a shared object is used).

ldd shows you the shared libraries an executable depends on. If Python has been built to use a shared object version of the python runtime will see libpythonXX.so in the list of shared object dependencies.

To build a version of Python that uses this shared object you should use the "--enable-shared" option when configuring. If successful, when you do ldd on python you will see libpythonXX.so in the output, and if you try the tests with clr built with PYTHON_WITHOUT_ENABLE_SHARED defined it will fail.

Building Python for static linking is something different. This is only useful for embedding Python into another executable without having dynamic link dependencies.

@den-run-ai
Copy link
Contributor Author

@tonyroberts great explanation, I will try to build with "--enable-shared" and do some testing.

denfromufa added 4 commits December 7, 2016 00:38
no more python 2.6 - missing subprocess.check_output()
# enable_shared = get_config_var("Py_ENABLE_SHARED")
# if enable_shared == 0:
defines.append("PYTHON_WITHOUT_ENABLE_SHARED")
enable_shared = get_config_var("Py_ENABLE_SHARED")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

capital "y" as in PY_ENABLE_SHARED?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@den-run-ai
Copy link
Contributor Author

@tonyroberts @filmor please review

@filmor filmor changed the title Update setup.py Work around a broken shared detection on some platforms Dec 7, 2016
@den-run-ai den-run-ai merged commit 66d755f into master Dec 7, 2016
@filmor filmor deleted the enable_shared_on_linux branch December 7, 2016 16:03
@@ -10,7 +10,7 @@
from distutils.spawn import find_executable
from distutils import log
from platform import architecture
from subprocess import Popen, CalledProcessError, PIPE, check_call
from subprocess import Popen, CalledProcessError, PIPE, check_call, check_output
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@denfromufa Did you try using _check_output instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but what is the difference?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the exact same thing 😄. Atleast it sounds like _check_output was copied from py27 so that earlier version (py26) could use it. I'm reviewing the changes we can make after dropping old python versions and noticed your code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants