Skip to content

Crashes on .NET Core 3 and 2 on Linux #967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
emilmuller opened this issue Oct 7, 2019 · 4 comments
Closed

Crashes on .NET Core 3 and 2 on Linux #967

emilmuller opened this issue Oct 7, 2019 · 4 comments

Comments

@emilmuller
Copy link

Can't make this work on Linux Mint 19 (Ubuntu 18.04).
Compiled ReleaseMonoPy3 with XPLAT;;PYTHON3;PYTHON37;UCS4;MONO_LINUX;PYTHON_WITH_PYMALLOC.
It seems to crash on Py.GIL():

using (Py.GIL())
{
    dynamic np = Py.Import("numpy");
    Console.WriteLine(np.cos(np.pi * 2));

    dynamic sin = np.sin;
    Console.WriteLine(sin(5));

    double c = np.cos(5) + sin(5);
    Console.WriteLine(c);

    dynamic a = np.array(new List<float> { 1, 2, 3 });
    Console.WriteLine(a.dtype);

    dynamic b = np.array(new List<float> { 6, 5, 4 }, dtype: np.int32);
    Console.WriteLine(b.dtype);

    Console.WriteLine(a * b);
    Console.ReadKey();
}

Upon further investigation, it seems to crash in InitializePlatformData(), as

IntPtr platformModule = PyImport_ImportModule("platform"); // Becomes NULL pointer

Running this line immediately after setting platformModule = ...

Console.WriteLine(new PythonException().Message);

yields:

ModuleNotFoundError : No module named '_posixsubprocess'

I also tried making sure the python home and path variables are correctly set:

PythonEngine.PythonHome = @"/usr/local/bin/python3.7";
PythonEngine.PythonPath = @"/usr/local/lib/python3.7";

I have no problem importing _posixsubprocess module when running the python3.7 binary from the terminal.
@emilmuller emilmuller mentioned this issue Oct 7, 2019
Closed
@filmor
Copy link
Member

filmor commented Oct 10, 2019

Please verify your installation. If you already get a PythonException, that means that libpython has been found and is loaded, so not finding modules is a pure configuration problem that has nothing to do with Python.NET and would affect any other embedding scenario as well. Check what python -c "import sys; print(sys.path) spits out and try setting that as your PythonPath (separated by : on Unix and ; on Windows).

@filmor filmor closed this as completed Oct 10, 2019
@Jeff17Robbins
Copy link
Contributor

Jeff17Robbins commented Dec 15, 2019 •
edited
Loading

@filmor I respectfully disagree about this issue having nothing to do with Python.NET. Please see my comment in #891.

I think the potential source of confusion is that the [DllImport] that finds and loads libpython does not set the RTLD_GLOBAL flag on its dlopen() call. And, furthermore, some Linuxes like Debian explicitly "does not link extensions to libpython".

[I assume that this is why runtime.cs takes such trouble to find out the OperatingSystemName and MachineName in InitializePlatformData, precisely in order to explictly load libpython with said RTLD_GLOBAL flag?]

So when platform.py changed to using subprocess.py, which on Linux attempts to import a C extension library _posixsubprocess, there's a crash.

This crash is, I think, precisely due to Python.NET running C Python code (importing platform.py) in a"semi-initialized state" inside its Initialize before it has done that explicit Load of libpython with RTLD_GLOBAL.

I don't think this crash reflects a configuration problem. I think this crash reflects on why runtime.cs needs to reload libpython in the first place!

For .NET Core 3.x, we could run this code instead of using platform.py, and get our required strings quicker, without this crash. For example, with code like this (tested on Debian 10 .NET Core 3.1):

            if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
                OperatingSystemName = "Windows";
            else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
                OperatingSystemName = "Darwin";
            else
                OperatingSystemName = "Linux";

            if (RuntimeInformation.ProcessArchitecture == Architecture.X86)
                MachineName = "x86";
            else if (RuntimeInformation.ProcessArchitecture == Architecture.X64)
                MachineName = "x86_64";
            else
                MachineName = "Other";

PS, I believe this analysis explains issue #946 also.

@Jeff17Robbins Jeff17Robbins mentioned this issue Dec 15, 2019
Closed
4 tasks
@filmor
Copy link
Member

filmor commented Dec 15, 2019

@Jeff17Robbins Well, Python.NET didn't decide to default to RTLD_LOCAL on .NET Core in DllImport (which I think is an unfortunate choice).

Nevertheless, you are completely right that this whole InitializePlatformData dance can be avoided by using RuntimeInformation (I think there is even a comment in our code that we could do that), but for this we need to "officially" update to .NET >4.7.1. There is a ticket for this (#897) and we'll decide this Tuesday, but as far as I can see everyone is in favour.

We wouldn't even need to assign our own values here anymore, we can just use the RuntimeInformation data directly.

@Jeff17Robbins
Copy link
Contributor

@filmor Conceivably, for .NET Core 3.1 anyway, we could override that unfortunate choice using System.Runtime.InteropServices.DllImportResolver. I'm not clear on how to get a call to this to run before all those [DllImport]s, but if you have any ideas about that, I'd be happy to give it a try. The resolver can call dlopen with the RTLD_GLOBAL flag.

It's not a solution for .NET Framework 4.7.x.

But it could tidy things up for .NET Core on Linux and remove the need for the second call to dlopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@filmor @Jeff17Robbins @emilmuller