Skip to content

python 3.7 issues #609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
den-run-ai opened this issue Jan 28, 2018 · 34 comments
Closed

python 3.7 issues #609

den-run-ai opened this issue Jan 28, 2018 · 34 comments

Comments

@den-run-ai
Copy link
Contributor

pycparser error:

https://travis-ci.org/pythonnet/pythonnet/jobs/334432678#L1991

pycparser.plyparser.ParseError: /usr/include/crypt.h:33:6: before: __THROW

@jonemo
Copy link

jonemo commented Apr 2, 2018

I am seeing the same error when attempting to build Docker images for https://github.com/jonemo/pythonnet-docker for all combinations that involve Python 3.6.5:

  • Python 3.6.5, Mono 4.8.0.524, Python.NET 2.3.0
  • Python 3.6.5, Mono 4.8.0.524, Python.NET master branch
  • Python 3.6.5, Mono 5.2.0.224, Python.NET master branch
  • Python 3.6.5, Mono 5.4.1.6, Python.NET master branch

Python 3.5.5 and 3.4.8 are not affected.

Here is the output I get for the command pip install pythonnet==2.3.0 (almost identical to what's behind the Travis link above):

Step 3/3 : RUN pip install pycparser   && pip install pythonnet==2.3.0
 ---> Running in 306362b2258a
Collecting pycparser
  Downloading pycparser-2.18.tar.gz (245kB)
Building wheels for collected packages: pycparser
  Running setup.py bdist_wheel for pycparser: started
  Running setup.py bdist_wheel for pycparser: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built pycparser
Installing collected packages: pycparser
Successfully installed pycparser-2.18
Collecting pythonnet==2.3.0
  Downloading pythonnet-2.3.0.tar.gz (1.5MB)
Building wheels for collected packages: pythonnet
  Running setup.py bdist_wheel for pythonnet: started
  Running setup.py bdist_wheel for pythonnet: finished with status 'error'
  Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zrnh3win/pythonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp9rlc9nv2pip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_ext
  Checking for updates from https://www.nuget.org/api/v2/.
  Currently running NuGet.exe 3.5.0.
  Updating NuGet.exe to 4.5.1.
  Update successful.
  MSBuild auto-detection: using msbuild version '14.0' from '/usr/lib/mono/xbuild/14.0/bin'.
  MSBUILD: error MSBUILD0004: Too many project files specified

  WARNING: Error reading msbuild project information, ensure that your input solution or project file is valid. NETCore and UAP projects will be skipped, only packages.config files will be restored.
  Restoring NuGet package NUnit.3.6.0.
  Restoring NuGet package NUnit.ConsoleRunner.3.6.0.
  Restoring NuGet package UnmanagedExports.1.2.7.
    GET https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.6.0/nunit.consolerunner.3.6.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit/3.6.0/nunit.3.6.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg
    OK https://api.nuget.org/v3-flatcontainer/nunit/3.6.0/nunit.3.6.0.nupkg 266ms
  Installing NUnit 3.6.0.
    OK https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg 910ms
  Installing UnmanagedExports 1.2.7.
    OK https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.6.0/nunit.consolerunner.3.6.0.nupkg 1216ms
  Installing NUnit.ConsoleRunner 3.6.0.
  Adding package 'UnmanagedExports.1.2.7' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Adding package 'NUnit.ConsoleRunner.3.6.0' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Added package 'UnmanagedExports.1.2.7' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Added package 'NUnit.ConsoleRunner.3.6.0' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Adding package 'NUnit.3.6.0' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Added package 'NUnit.3.6.0' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'

  NuGet Config files used:
      /root/.config/NuGet/NuGet.Config

  Feeds used:
      https://api.nuget.org/v3/index.json

  Installed:
      3 package(s) to packages.config projects
  clang: warning: /usr/local/include/python3.6m: 'linker' input unused
  Traceback (most recent call last):
    File "tools/geninterop/geninterop.py", line 292, in <module>
      sys.exit(main())
    File "tools/geninterop/geninterop.py", line 274, in main
      ast = parser.parse(python_h)
    File "/usr/local/lib/python3.6/site-packages/pycparser/c_parser.py", line 152, in parse
      debug=debuglevel)
    File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 331, in parse
      return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
    File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
      tok = call_errorfunc(self.errorfunc, errtoken, self)
    File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
      r = errorfunc(token)
    File "/usr/local/lib/python3.6/site-packages/pycparser/c_parser.py", line 1761, in p_error
      column=self.clex.find_tok_column(p)))
    File "/usr/local/lib/python3.6/site-packages/pycparser/plyparser.py", line 66, in _parse_error
      raise ParseError("%s: %s" % (coord, msg))
  pycparser.plyparser.ParseError: /usr/include/crypt.h:33:6: before: __THROW
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-zrnh3win/pythonnet/setup.py", line 405, in <module>
      zip_safe=False,
    File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 204, in run
      self.run_command('build')
    File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 339, in run
      self.build_extensions()
    File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
      self._build_extensions_serial()
    File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-build-zrnh3win/pythonnet/setup.py", line 191, in build_extension
      subprocess.check_call([sys.executable, geninterop, interop_file])
    File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/usr/local/bin/python', 'tools/geninterop/geninterop.py', 'src/runtime/interop36m.cs']' returned non-zero exit status 1.

  ----------------------------------------
  Failed building wheel for pythonnet
  Running setup.py clean for pythonnet
Failed to build pythonnet
Installing collected packages: pythonnet
  Running setup.py install for pythonnet: started
    Running setup.py install for pythonnet: finished with status 'error'
    Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zrnh3win/pythonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9h9w6f8b-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    Checking for updates from https://www.nuget.org/api/v2/.
    Currently running NuGet.exe 4.5.1.
    NuGet.exe is up to date.
    MSBuild auto-detection: using msbuild version '14.0' from '/usr/lib/mono/xbuild/14.0/bin'.
    MSBUILD: error MSBUILD0004: Too many project files specified

    WARNING: Error reading msbuild project information, ensure that your input solution or project file is valid. NETCore and UAP projects will be skipped, only packages.config files will be restored.
    All packages listed in packages.config are already installed.
    clang: warning: /usr/local/include/python3.6m: 'linker' input unused
    Traceback (most recent call last):
      File "tools/geninterop/geninterop.py", line 292, in <module>
        sys.exit(main())
      File "tools/geninterop/geninterop.py", line 274, in main
        ast = parser.parse(python_h)
      File "/usr/local/lib/python3.6/site-packages/pycparser/c_parser.py", line 152, in parse
        debug=debuglevel)
      File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 331, in parse
        return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
      File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
        tok = call_errorfunc(self.errorfunc, errtoken, self)
      File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
        r = errorfunc(token)
      File "/usr/local/lib/python3.6/site-packages/pycparser/c_parser.py", line 1761, in p_error
        column=self.clex.find_tok_column(p)))
      File "/usr/local/lib/python3.6/site-packages/pycparser/plyparser.py", line 66, in _parse_error
        raise ParseError("%s: %s" % (coord, msg))
    pycparser.plyparser.ParseError: /usr/include/crypt.h:33:6: before: __THROW
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-zrnh3win/pythonnet/setup.py", line 405, in <module>
        zip_safe=False,
      File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-build-zrnh3win/pythonnet/setup.py", line 191, in build_extension
        subprocess.check_call([sys.executable, geninterop, interop_file])
      File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/usr/local/bin/python', 'tools/geninterop/geninterop.py', 'src/runtime/interop36m.cs']' returned non-zero exit status 1.

    ----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zrnh3win/pythonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9h9w6f8b-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-zrnh3win/pythonnet/
The command '/bin/sh -c pip install pycparser   && pip install pythonnet==2.3.0' returned a non-zero code: 1

@qherreros
Copy link

Same issue here with python3.6.5 on a clean pipenv environment.

@stonebig
Copy link
Contributor

stonebig commented May 27, 2018

any hope of Windows pythonnet wheel under Python-3.7 ? where a few days before RC.

@fractus
Copy link
Contributor

fractus commented Jun 2, 2018

I have the same issue on the Windows Linux subsystem using (Python, Mono, Dotnet) = (3.6.5, 5.12, 2.1.300) building out of master. However, you can "cheat" to make it work until a fix is in place, as follows:

  • Create a conda env on a Python version that works (I used 3.6.2)
  • Build using python setup.py bdist_wheel --xplat
  • Install from you root (3.6.5) environment using pip install --no-index --find-links=.\dist\ pythonnet

(instruction from Installation page)

@den-run-ai
Copy link
Contributor Author

@fractus @jonemo so which python 3.6.x version broke this pycparser parsing? then i can probably send this over to cpython bug tracker.

@den-run-ai
Copy link
Contributor Author

also if someone can attach debug files generated by pycparser/ply during this error, that would be helpful too.

@fractus
Copy link
Contributor

fractus commented Jun 3, 2018

It appears to be 3.6.5 -- it works on 3.6.{2, 3, 4}. On master, it fails on l. 220 of setup.py; to replicate from the commad line:

python tools/geninterop/geninterop.py src/runtime/interop36m.cs

Not sure how to generate the debug files of pycparser.

Ps. This is run in conda environments (conda create -n c36# python=3.6.#) plus pycparser 2.18.

@den-run-ai
Copy link
Contributor Author

@fractus it appears that this additional header crypt.h was added in python 3.6.5 (backported) and we may need to either force exclude it in geniterop.py or add a fake clib include for it to pycparser. It is probably not c99 compliant header.

https://docs.python.org/3/whatsnew/changelog.html#id8
https://github.com/python/cpython/pull/5284/files
https://eli.thegreenplace.net/2015/on-parsing-c-type-declarations-and-fake-headers/

@thomasmortiz
Copy link

thomasmortiz commented Jun 10, 2018

@denfromufa Not sure if this is what you're looking for, but I've attached a trace log file generated when trying to build pythonnet 2.3.0 from source on Pop!_OS 18.04 LTS (Linux x64). I obtained the source from https://anaconda.org/pythonnet/pythonnet/files . I get the same errors noted above.
trace.log.gz
Python -V = Python 3.6.5 :: Anaconda custom (64-bit)

@den-run-ai
Copy link
Contributor Author

@thomasmortiz that trace does not contain the pycparser error. this is probably only tracing of python imports. actually we don't need any debug/log, just fix/avoid crypt.h parsing.

@stonebig
Copy link
Contributor

stonebig commented Jul 1, 2018

Python-3.7.0 is out.

@den-run-ai
Copy link
Contributor Author

@stonebig waiting for this 😸 appveyor/ci#2475

I added a py37 branch in the meantime with interop37.cs copied from interop36.cs as-is.

#698

@stonebig
Copy link
Contributor

appveyor seems ready, now.

@stonebig
Copy link
Contributor

stonebig commented Jul 22, 2018

the wheel pythonnet-2.4.0.dev0-cp37-cp37m-win32.whl from cgohlke works great on Python-3.7 !

@CaerphillyMediaLtd
Copy link

This also occurs for me on Python 3.5.2

@den-run-ai
Copy link
Contributor Author

@stonebig those wheels from @cgohlke may fail on some tests according to Travis CI (see PR #698)

https://www.lfd.uci.edu/~gohlke/pythonlibs/#pythonnet

@stonebig
Copy link
Contributor

stonebig commented Jul 24, 2018

still a huge improvement over nothing, and it passes my stupid test.

@den-run-ai
Copy link
Contributor Author

some progress on the python 3.7 support, but now an error from clang during parsing:

#698 (comment)

I have no idea how to fix this!

@stonebig
Copy link
Contributor

On the problem related to https://bugs.python.org/issue31370, maybe @pitrou could give you a quick suggestion. Let's pray.

@pitrou
Copy link

pitrou commented Aug 14, 2018

The only suggestion is to build Python on a platform that has support for threads (either pthreads, or Win32).

@den-run-ai
Copy link
Contributor Author

den-run-ai commented Aug 14, 2018

@pitrou thanks for stepping in, but this is ubuntu trusty/xenial and normal cpython 2.7-3.7 from travis ci with obvious support of threads. So do I need to pass some special flag to clang in order to parse this header /opt/python/3.7.0/include/python3.7m/pythread.h:122:5?

https://travis-ci.org/pythonnet/pythonnet/jobs/415741697#L1999

@pitrou
Copy link

pitrou commented Aug 14, 2018

Well, you probably know better than me what this build chain does. But just be aware that Python 3.7 compiles fine in CPython's own Travis-CI setup ;-)

At least the command line looks unusual ('-I', 'tools/geninterop/fake_libc_include'?):
https://travis-ci.org/pythonnet/pythonnet/jobs/415741697#L2017

@den-run-ai
Copy link
Contributor Author

den-run-ai commented Aug 14, 2018 via email

@den-run-ai
Copy link
Contributor Author

ok, reported in pycparser: eliben/pycparser#276

@den-run-ai
Copy link
Contributor Author

Maybe we should try out this pycparser extended version or/and gcc parser:

https://github.com/inducer/pycparserext

#510

@den-run-ai
Copy link
Contributor Author

This is now fixed, like @eliben suggested it was a missing #define to the compiler:

https://github.com/pythonnet/pythonnet/blob/py37/tools/geninterop/geninterop.py#L177

"-D _POSIX_THREADS"

@den-run-ai
Copy link
Contributor Author

Next is nuget/msbuild issue under Mono:

NuGet/Home#7202

@den-run-ai
Copy link
Contributor Author

Python 3.7 support is merged!

#698

@den-run-ai
Copy link
Contributor Author

oh, and this fixes reported parsing errors on Python 3.6+ with geninterop/pycparser/ply.

@Jeff17Robbins
Copy link
Contributor

The pythonnet VS2017 GUI build fails for Python 3.7. Apparently due to not have interop37.cs in Python.Runtime.csproj (but I'm not a Visual Studio expert):

  <ItemGroup Condition=" '$(PythonInteropFile)' == '' ">
    <Compile Include="interop27.cs" />
    <Compile Include="interop34.cs" />
    <Compile Include="interop35.cs" />
    <Compile Include="interop36.cs" />
  </ItemGroup>

@den-run-ai
Copy link
Contributor Author

den-run-ai commented Oct 27, 2018

@Jeff17Robbins my bad, can you please submit a PR to include this file to project file?

@Jeff17Robbins
Copy link
Contributor

@denfromufa I'll get help from some actual programmers at work on this. I'm not too git savvy.

@den-run-ai
Copy link
Contributor Author

some follow-up issues:

#720

@Jeff17Robbins you can just make this change from github UI without git.

@Jeff17Robbins
Copy link
Contributor

@denfromufa @paulie4 and I discussed, and he created the PR. As he already stated, we are only able to test with VS2017.

Also already stated, the UnmanagedExports tool doesn't seem to know how to find lib.exe in the VS2017 folder tree, and so it creates a pythonnet that doesn't actually work. Which can be really confusing!

The Release Notes on the nuget site says

  • no longer fails the build when it can't find lib.exe

But...that's not a good thing!

The failure shows up at runtime, not build time:

import clr

ImportError: dynamic module does not define init function (initclr)

We couldn't find the source code site for that UnmanagedExports tool, so we don't know how to fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants