Skip to content

How to install QuantConnect's version of pythonnet? #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
hsm207 opened this issue Jun 14, 2020 · 5 comments
Closed

How to install QuantConnect's version of pythonnet? #43

hsm207 opened this issue Jun 14, 2020 · 5 comments

Comments

@hsm207
Copy link

hsm207 commented Jun 14, 2020

Environment

Details

  • Describe what you were trying to get done.

    I am trying to install pythonnet inside the lean docker container.

  • What commands did you run to trigger this issue? If you can provide a
    Minimal, Complete, and Verifiable example
    this will help us understand the issue.

docker run --rm \
    --entrypoint /bin/bash \
    quantconnect/lean:latest -c "pip install git+git://github.com/QuantConnect/pythonnet.git"
  • If there was a crash, please include the traceback here.
  Collecting git+git://github.com/QuantConnect/pythonnet.git
  Cloning git://github.com/QuantConnect/pythonnet.git to /tmp/pip-req-build-pr2g_nr8
  Running command git clone -q git://github.com/QuantConnect/pythonnet.git /tmp/pip-req-build-pr2g_nr8
Building wheels for collected packages: pythonnet
  Building wheel for pythonnet (setup.py): started
  Building wheel for pythonnet (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/miniconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-26m73jmp
       cwd: /tmp/pip-req-build-pr2g_nr8/
  Complete output (104 lines):
  running bdist_wheel
  running build
  running build_ext
  Checking for updates from https://www.nuget.org/api/v2/.
  Currently running NuGet.exe 4.1.0.
  Updating NuGet.exe to 5.5.1.
  Update successful.
  MSBuild auto-detection: using msbuild version '15.0' from '/usr/lib/mono/msbuild/15.0/bin'.
  Restoring NuGet package UnmanagedExports.1.2.7.
  Restoring NuGet package NUnit.ConsoleRunner.3.7.0.
  Restoring NuGet package NUnit.3.7.1.
    GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg
    GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg
    GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg
    OK https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg 26ms
  Installing UnmanagedExports 1.2.7.
    OK https://api.nuget.org/v3-flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg 157ms
  Installing NUnit 3.7.1.
    OK https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg 193ms
  Installing NUnit.ConsoleRunner 3.7.0.
    OK https://api.nuget.org/v3-flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg 512ms
    NotFound https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg 1436ms
    OK https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg 1463ms
  Adding package 'UnmanagedExports.1.2.7' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
  Adding package 'NUnit.ConsoleRunner.3.7.0' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
    NotFound https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg 1707ms
  Added package 'UnmanagedExports.1.2.7' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
  Added package 'NUnit.ConsoleRunner.3.7.0' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
  Adding package 'NUnit.3.7.1' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
  Added package 'NUnit.3.7.1' to folder '/tmp/pip-req-build-pr2g_nr8/packages'

  NuGet Config files used:
      /tmp/pip-req-build-pr2g_nr8/NuGet.config
      /root/.config/NuGet/NuGet.Config

  Feeds used:
      https://dotnet.myget.org/F/dotnet-core/api/v3/index.json
      https://api.nuget.org/v3/index.json

  Installed:
      3 package(s) to packages.config projects
  sh: umask: I/O error
  clang: warning: /opt/miniconda3/include/python3.6m: 'linker' input unused
  Traceback (most recent call last):
    File "tools/geninterop/geninterop.py", line 292, in <module>
      sys.exit(main())
    File "tools/geninterop/geninterop.py", line 274, in main
      ast = parser.parse(python_h)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/c_parser.py", line 152, in parse
      debug=debuglevel)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 331, in parse
      return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
      tok = call_errorfunc(self.errorfunc, errtoken, self)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
      r = errorfunc(token)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/c_parser.py", line 1861, in p_error
      column=self.clex.find_tok_column(p)))
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/plyparser.py", line 67, in _parse_error
      raise ParseError("%s: %s" % (coord, msg))
  pycparser.plyparser.ParseError: /usr/include/crypt.h:33:6: before: __THROW
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-pr2g_nr8/setup.py", line 527, in <module>
      zip_safe=False,
    File "/opt/miniconda3/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/miniconda3/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/miniconda3/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/opt/miniconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-pr2g_nr8/setup.py", line 475, in run
      return bdist_wheel.bdist_wheel.run(self)
    File "/opt/miniconda3/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/opt/miniconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/miniconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/miniconda3/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/miniconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/miniconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 339, in run
      self.build_extensions()
    File "/opt/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
      self._build_extensions_serial()
    File "/opt/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-req-build-pr2g_nr8/setup.py", line 220, in build_extension
      subprocess.check_call([sys.executable, geninterop, interop_file])
    File "/opt/miniconda3/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/opt/miniconda3/bin/python', 'tools/geninterop/geninterop.py', 'src/runtime/interop36m.cs']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for pythonnet
  Running setup.py clean for pythonnet
Failed to build pythonnet
Installing collected packages: pythonnet
    Running setup.py install for pythonnet: started
    Running setup.py install for pythonnet: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /opt/miniconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-y0rg3pnr/install-record.txt --single-version-externally-managed --compile --install-headers /opt/miniconda3/include/python3.6m/pythonnet
         cwd: /tmp/pip-req-build-pr2g_nr8/
    Complete output (6 lines):
    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help

    error: option --single-version-externally-managed not recognized
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/miniconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-y0rg3pnr/install-record.txt --single-version-externally-managed --compile --install-headers /opt/miniconda3/include/python3.6m/pythonnet Check the logs for full command output.

Installing from the original repo works. This command:

docker run --rm \
    --entrypoint /bin/bash \
    quantconnect/lean:latest -c "pip install git+git://github.com/pythonnet/pythonnet.git@v2.4.0"

returns:

Collecting git+git://github.com/pythonnet/pythonnet.git@v2.4.0
  Cloning git://github.com/pythonnet/pythonnet.git (to revision v2.4.0) to /tmp/pip-req-build-y7hjxlom
  Running command git clone -q git://github.com/pythonnet/pythonnet.git /tmp/pip-req-build-y7hjxlom
  Running command git checkout -q cc538f6a5deedc7af9ec6ab9d3ea0d22a460b54a
Building wheels for collected packages: pythonnet
  Building wheel for pythonnet (setup.py): started
  Building wheel for pythonnet (setup.py): finished with status 'done'
  Created wheel for pythonnet: filename=pythonnet-2.4.0-cp36-cp36m-linux_x86_64.whl size=85671 sha256=a36565ffde5514e69933089564d0c593303f1358c7a506c2a3a1c78203fa6068
  Stored in directory: /tmp/pip-ephem-wheel-cache-2gtul90l/wheels/14/b9/ea/3547a0e9a89fef0a8c5fddb2519d98a89eae9b2b1d00829498
Successfully built pythonnet
Installing collected packages: pythonnet
Successfully installed pythonnet-2.4.0
@jaredbroad
Copy link
Member

Hi @hsm207 it is already installed in that container. Mind sharing more detail on what you're trying to achieve?

@hsm207
Copy link
Author

hsm207 commented Jun 14, 2020

@jaredbroad I'm trying to develop algorithms locally using Python and VS Code.

Based on the content of DockerfileLeanFoundation, I can see that the python interpreter being used is located at /opt/miniconda3/bin/python inside the quantconnect/lean image. But if I launch that interpreter and execute:

import clr

I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'clr'

This made me think that pythonnet is not installed.

But it is interesting to note that if I run the interpreter with mono /Lean/Launcher/bin/Debug/nPython.exe then I can successfully run import clr. Unfortunately, this way of starting the Python interpreter is not supported in VS Code.

Do you have any recommendations on how people can get started developing algorithms in Python locally? I'm open to using any IDE as long I can develop in a containerized environment.

@hsm207
Copy link
Author

hsm207 commented Jun 16, 2020

Hi @hsm207 it is already installed in that container.

@jaredbroad I don't think it's already installed in that container. Running pip freeze | grep pythonnet in that container does not return any results.

@Martin-Molinero
Copy link
Member

Hey @hsm207,
You are right, pythonNet isn't installed but it is present. Due to pythonNet issues, currently, for linux, we have two different binaries for different use case scenarios: C# calling PY (default for running Launcher/QCAlgorithms) and Py calling C#. Believe the use case you want is Py->C#, which matches the use case of Research since that case isn't the default it requires copying over the right clr/pythonNet binary can be seen in the previous link. Maybe you could test directly using the research docker image for this use case quantconnect/research. For running the algorithm, through the launcher, quantconnect/lean should be used. I hope this helps!

@hsm207
Copy link
Author

hsm207 commented Jun 17, 2020

@Martin-Molinero thanks for the suggestion. The research docker image was exactly what I needed.

@hsm207 hsm207 closed this as completed Jun 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jaredbroad @hsm207 @Martin-Molinero