Skip to content

Using pythonnet in docker container #303

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AlexCatarino opened this issue Dec 7, 2016 · 19 comments
Closed

Using pythonnet in docker container #303

AlexCatarino opened this issue Dec 7, 2016 · 19 comments
Milestone

Comments

@AlexCatarino
Copy link
Contributor

We are trying to support algorithms written in python in QuantConnect/Lean (which is written in C#).
So far, we were able to accomplish that locally in Windows. However, we have to make it work in a docker container in order to launch algorithms in our servers, but we could't make pythonnet work in that environment.

Here are the steps we are following:

  • Run a container:
    docker run -it quantconnect/lean:foundation /bin/bash
  • Please checkout what system configuration quantconnect/lean:foundation has here.
  • Install python:
apt-get update 
apt-get install -y libglib2.0-dev python-all-dev python-pip git 
pip install --upgrade pip && pip install pandas scipy numpy
  • Install pythonnet
    pip install git+https://github.com/pythonnet/pythonnet

After that, we ran the three winforms demos. All throw the same exception:

 System.TypeInitializationException: The type initializer for 'System.Windows.Forms.WindowsFormsSynchronizationContext' threw an exception.
  at System.Windows.Forms.Control..ctor () [0x00014] in <5d999439b3404adfac86e7eebb5f00d6>:0
  at System.Windows.Forms.ScrollableControl..ctor () [0x00000] in <5d999439b3404adfac86e7eebb5f00d6>:0
  at System.Windows.Forms.ContainerControl..ctor () [0x0000e] in <5d999439b3404adfac86e7eebb5f00d6>:0
  at System.Windows.Forms.Form..ctor () [0x00012] in <5d999439b3404adfac86e7eebb5f00d6>:0
  at (wrapper managed-to-native) System.Reflection.MonoCMethod:InternalInvoke (System.Reflection.MonoCMethod,object,object[],System.Exception&)
  at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00002] in <dca3b561b8ad4f9fb10141d81b39ff45>:0

Finally, we ran the example of embedding Python in .NET from pythonnet README:
mcs /reference:Python.Runtime.dll test.cs && mono test.exe
that yields the following exception:

Unhandled Exception:
System.EntryPointNotFoundException: Py_IsInitialized
  at (wrapper managed-to-native) Python.Runtime.Runtime:Py_IsInitialized ()
  at Python.Runtime.Runtime.Initialize () [0x0000d] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Python.Runtime.PythonEngine.Initialize () [0x00014] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Python.Runtime.Py.GIL () [0x0000a] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Test.Test.Main (System.String[] args) [0x00000] in <d894aae141fa4a98b826174854b8bf77>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.EntryPointNotFoundException: Py_IsInitialized
  at (wrapper managed-to-native) Python.Runtime.Runtime:Py_IsInitialized ()
  at Python.Runtime.Runtime.Initialize () [0x0000d] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Python.Runtime.PythonEngine.Initialize () [0x00014] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Python.Runtime.Py.GIL () [0x0000a] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Test.Test.Main (System.String[] args) [0x00000] in <d894aae141fa4a98b826174854b8bf77>:0
@den-run-ai
Copy link
Contributor

@AlexCatarino

  • winforms is not available on linux.

  • can you try with latest master of pythonnet? we fixed some issues with wrong libpython settings in some linux distros.

  • can you replicate the settings from .travis.yml? particularly use mono version 4.2.4.4, due to bugs in later and previous versions.

@AlexCatarino
Copy link
Contributor Author

I am using the last master, when I install pythonnet using pip+git, right?
pip install git+https://github.com/pythonnet/pythonnet
Let me try mono 4.2.44! Thanks.

@den-run-ai
Copy link
Contributor

den-run-ai commented Dec 8, 2016 via email

@den-run-ai
Copy link
Contributor

I apologize, WinForms is available on Mono, but I have not tested this:

http://www.mono-project.com/docs/gui/winforms/

@AlexCatarino
Copy link
Contributor Author

AlexCatarino commented Dec 8, 2016

From my base image: docker run -it phusion/baseimage:0.9.19 /bin/bash
I have replicated the setting from .travis.yml. It did pass the test:

> python src/tests/runtests.py
Load clr import hook
.............................................................Python.Test.SubclassConstructorTest
..........................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 343 tests in 2.956s

OK

However, when I ran the example of embedding Python in .NET from pythonnet README, I got the same exception described above.

@jaredbroad
Copy link

@denfromufa any thoughts on how to fix this? Its blocking using Python.Net in our platform / https://github.com/QuantConnect/Lean (It works fine in windows)

@den-run-ai
Copy link
Contributor

den-run-ai commented Dec 21, 2016 via email

@AlexCatarino
Copy link
Contributor Author

QuantConnect/Lean is able to support python in Linux with pythonnet!

In order to do it, we needed to discarded changes made by #300. Since ldd does not show libpython, pythonnet is compiled with the PYTHON_WITHOUT_ENABLE_SHARED flag that results in the reported exception. On the other hand, without that flag, import clr fails.

However, if we get Python.Runtime.dll, Python.Runtime.dll.config, clr.so, create a dynamic link to the library (ln -s /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so .) and set the environment to the directory these files are found, we are able to embed Python in .NET:

$ mcs test.cs /r:Python.Runtime.dll
$ mono test.exe
$ 1/3/2017 11:30:57 PM 3.14159265359

Files:

// test.cs
namespace TEST
{
   using System;
   using System.IO;
   using Python.Runtime;
   class TEST
   {
       public static void Main(string[] args)
       {
           Environment.SetEnvironmentVariable("PYTHONPATH", Environment.CurrentDirectory);
           PythonEngine.Initialize();

           try
           {
               using (Py.GIL())
               {
                   dynamic clrModule = Py.Import("pytest");
                   clrModule.print_now();
               }
           }
           finally
           {
               PythonEngine.Shutdown();
           }
       }
   }
}
# pytest.py
import clr
import numpy as np
clr.AddReference("System")
import System

def print_now():
    print System.DateTime.Now, np.pi

@den-run-ai
Copy link
Contributor

@AlexCatarino this is good finding! would you mind putting a pull request for this?

@AlexCatarino
Copy link
Contributor Author

AlexCatarino commented Jan 6, 2017

My "fix" is basically to undo #300. It would break what #300 fixes...
I am thinking that putting a compilation flag could be a proper fix.

@den-run-ai
Copy link
Contributor

den-run-ai commented Jan 6, 2017 via email

@den-run-ai den-run-ai added this to the 2.3.0 milestone Feb 2, 2017
@den-run-ai
Copy link
Contributor

@AlexCatarino @jaredbroad I can confirm your problem and this is also affecting our testing on Travis CI. Let me look for solutions. The worst we may have to do is to recompile CPython with the correct flags or find Python dsitribution that uses these settings.

@den-run-ai
Copy link
Contributor

@AlexCatarino @jaredbroad I tried Miniconda 3.5 64bit on Linux and it works in both directions - embedding and extending pythonnet!

Can you use Miniconda or Anaconda in your docker images?

@vmuriart
Copy link
Contributor

vmuriart commented Feb 5, 2017

@denfromufa I think this item from the old TODO document might be related. I don't know if it helps.

Let's talk to the Debian and Ubuntu maintainers for Python in order to convince
them to build Python with --enable-shared. Ubuntu's Python is semi static and not
linked against libpython2.x.so. This causes trouble with clr.so.

@den-run-ai
Copy link
Contributor

so @vmuriart found a solution to this problem that works with system python, no miniconda necessary:

#245 (comment)

@vmuriart
Copy link
Contributor

vmuriart commented Mar 3, 2017

@AlexCatarino were you able to resolve your issue? There's a couple docker images/recipes on https://hub.docker.com/r/pythonnet/pythonnet/tags/

@AlexCatarino
Copy link
Contributor Author

We have resolved our issue and will keep an eye on those images/recipes.
Thank you very much!

@vmuriart vmuriart closed this as completed Mar 6, 2017
@CoFlows
Copy link

CoFlows commented Jan 13, 2018

Hello gents, I need help with python3.6...

Thank you so much for sharing the conversation. I am trying to run this with Miniconda 3.6 but I get the following error:
Unhandled Exception: System.Exception: dlsym: dotnet: undefined symbol: _PyObject_NextNotImplemented
at Python.Runtime.NativeMethods.GetProcAddress(IntPtr dllHandle, String name) in runtime.cs:line 69
at Python.Runtime.Runtime.Initialize() in runtime.cs:line 322
at Python.Runtime.PythonEngine.Initialize(IEnumerable`1 args, Boolean setSysArgv) in pythonengine.cs:line 168
at Python.Runtime.Py.GIL() in pythonengine.cs:line 547

Building the docker:
RUN apt-get -qq update && apt-get -qq -y install curl bzip2
#&& curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh
&& bash /tmp/miniconda.sh -bfp /usr/local
&& rm -rf /tmp/miniconda.sh
&& conda install -y python=3
&& conda update conda
&& apt-get -qq -y remove curl bzip2
&& apt-get -qq -y autoremove
&& apt-get autoclean
&& rm -rf /var/lib/apt/lists/* /var/log/dpkg.log
&& conda clean --all --yes

RUN ln -s /lib/x86_64-linux-gnu/libdl.so.2 libdl.so
RUN ln -s /usr/local/lib/libpython3.6m.so .

python -V works and give me 3.6.4 as expected.

PS: Miniconda 2.7 doesn't work either....
what does work is:
apt-get update
apt-get install -y libglib2.0-dev python-all-dev python-pip git
pip install --upgrade pip && pip install pandas scipy numpy

Any ideas of how to make work? I need anything that makes python3.6 work...
THANKS

@CoFlows
Copy link

CoFlows commented Jan 14, 2018

Ok, I found a clue, the problem seems to be specific to how Anaconda is installed on a Debian/Stretch Docker.
I get this to work with the python coming from debian packages. Even python 3.5 works when using the apt-get python3 command.

Anaconda doesn't work for me neither 2 or 3.....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants