Skip to content

Commit 9d2be14

Browse files
filmorfilmor
committed
Replace clr module by loading through clr_loader
clr_loader is based on CFFI and allows loading pythonnet on different .NET runtime implementations, currently .NET Framework, .NET Core and Mono. Apart from dropping the respective old code, this requires the following changes: - Move libpython discovery into Python by vendoring and adjusting `find_libpython` - Adjust the GIL handling in the startup code as CFFI releases the GIL when calling the external function - Remove the intermittent `configure` command as it is not required anymore - Adjust a few test-cases - Remove `_AtExit` Due to issues with the reference counts, the `atexit` callback is currently essentially a no-op until these are fixed.
1 parent 6b2347a commit 9d2be14

28 files changed

+804
-824
lines changed

‎.github/workflows/main.yml

Lines changed: 12 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -12,33 +12,9 @@ jobs:
1212
fail-fast: false
1313
matrix:
1414
os: [windows, ubuntu, macos]
15-
pyver_minor: [6, 7, 8, 9]
15+
python: ["3.6", "3.7", "3.8", "3.9"]
1616
platform: [x64]
1717
shutdown_mode: [Normal, Soft]
18-
include:
19-
- os: ubuntu
20-
pyver_minor: 6
21-
dll_suffix: m
22-
- os: ubuntu
23-
pyver_minor: 7
24-
dll_suffix: m
25-
26-
- os: macos
27-
dll_prefix: lib
28-
dll_pyver_major: '3.'
29-
dll_suffix: m
30-
- os: ubuntu
31-
dll_prefix: lib
32-
dll_pyver_major: '3.'
33-
- os: windows
34-
dll_pyver_major: '3'
35-
36-
- os: ubuntu
37-
dll_ext: .so
38-
- os: windows
39-
dll_ext: .dll
40-
- os: macos
41-
dll_ext: .dylib
4218

4319
env:
4420
PYTHONNET_SHUTDOWN_MODE: ${{ matrix.SHUTDOWN_MODE }}
@@ -56,10 +32,10 @@ jobs:
5632
- name: Setup .NET
5733
uses: actions/setup-dotnet@v1
5834

59-
- name: Set up Python 3.${{ matrix.pyver_minor }}
35+
- name: Set up Python ${{ matrix.python }}
6036
uses: actions/setup-python@v2
6137
with:
62-
python-version: 3.${{ matrix.pyver_minor }}
38+
python-version: ${{ matrix.python }}
6339
architecture: ${{ matrix.platform }}
6440

6541
- name: Install dependencies
@@ -68,31 +44,26 @@ jobs:
6844
6945
- name: Build and Install
7046
run: |
71-
python setup.py configure
7247
pip install -v .
7348
74-
# TODO this should be gone once clr module sets PythonDLL or preloads it
75-
- name: Python Tests
76-
run: pytest
77-
if: ${{ matrix.os != 'macos' }}
78-
env:
79-
PYTHONNET_PYDLL: ${{ matrix.DLL_PREFIX }}python${{matrix.DLL_PYVER_MAJOR}}${{matrix.PYVER_MINOR}}${{matrix.DLL_SUFFIX}}${{matrix.DLL_EXT}}
49+
- name: Set Python DLL path (non Windows)
50+
if: ${{ matrix.os != 'windows' }}
51+
run: |
52+
python -m pythonnet.find_libpython --export >> $GITHUB_ENV
8053
54+
- name: Set Python DLL path (Windows)
55+
if: ${{ matrix.os == 'windows' }}
56+
run: |
57+
python -m pythonnet.find_libpython --export | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
58+
8159
- name: Python Tests
8260
run: pytest
83-
if: ${{ matrix.os == 'macos' }}
8461

8562
- name: Embedding tests
8663
run: dotnet test --runtime any-${{ matrix.platform }} src/embed_tests/
87-
if: ${{ matrix.os != 'macos' }} # Not working right now, doesn't find libpython
88-
env:
89-
PYTHONNET_PYDLL: ${{ matrix.DLL_PREFIX }}python${{matrix.DLL_PYVER_MAJOR}}${{matrix.PYVER_MINOR}}${{matrix.DLL_SUFFIX}}${{matrix.DLL_EXT}}
9064

9165
- name: Python tests run from .NET
9266
run: dotnet test --runtime any-${{ matrix.platform }} src/python_tests_runner/
93-
if: ${{ matrix.os == 'windows' }} # Not working for others right now
94-
env:
95-
PYTHONNET_PYDLL: ${{ matrix.DLL_PREFIX }}python${{matrix.DLL_PYVER_MAJOR}}${{matrix.PYVER_MINOR}}${{matrix.DLL_SUFFIX}}${{matrix.DLL_EXT}}
9667

9768
# TODO: Run perf tests
9869
# TODO: Run mono tests on Windows?

‎.gitignore

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
11
/src/runtime/interopNative.cs
22

3-
# Configuration data
4-
configured.props
5-
63
# General binaries and Build results
74
*.dll
85
*.exe

‎Directory.Build.props

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,4 @@
1818
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
1919
</PackageReference>
2020
</ItemGroup>
21-
<Import Project="$(MSBuildThisFileDirectory)configured.props" Condition="Exists('$(MSBuildThisFileDirectory)configured.props')" />
2221
</Project>

‎appveyor.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,9 @@ environment:
3030
init:
3131
# Update Environment Variables based on matrix/platform
3232
- set PY_VER=%PYTHON_VERSION:.=%
33-
- set PYTHONNET_PYDLL=python%PY_VER%.dll
3433
- set PYTHON=C:\PYTHON%PY_VER%
3534
- if %PLATFORM%==x64 (set PYTHON=%PYTHON%-x64)
35+
- set PYTHONNET_PYDLL=%PYTHON%\python%PY_VER%.dll
3636

3737
# Put desired Python version first in PATH
3838
- set PATH=%PYTHON%;%PYTHON%\Scripts;%PATH%
@@ -42,7 +42,6 @@ install:
4242
- pip install --upgrade -r requirements.txt --quiet
4343

4444
build_script:
45-
- python setup.py configure
4645
# Create clean `sdist`. Only used for releases
4746
- python setup.py --quiet sdist
4847
- python setup.py bdist_wheel

‎clr.py

Lines changed: 2 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -2,40 +2,5 @@
22
Legacy Python.NET loader for backwards compatibility
33
"""
44

5-
def _get_netfx_path():
6-
import os, sys
7-
8-
if sys.maxsize > 2 ** 32:
9-
arch = "amd64"
10-
else:
11-
arch = "x86"
12-
13-
return os.path.join(os.path.dirname(__file__), "pythonnet", "netfx", arch, "clr.pyd")
14-
15-
16-
def _get_mono_path():
17-
import os, glob
18-
19-
paths = glob.glob(os.path.join(os.path.dirname(__file__), "pythonnet", "mono", "clr.*so"))
20-
return paths[0]
21-
22-
23-
def _load_clr():
24-
import sys
25-
from importlib import util
26-
27-
if sys.platform == "win32":
28-
path = _get_netfx_path()
29-
else:
30-
path = _get_mono_path()
31-
32-
del sys.modules[__name__]
33-
34-
spec = util.spec_from_file_location("clr", path)
35-
clr = util.module_from_spec(spec)
36-
spec.loader.exec_module(clr)
37-
38-
sys.modules[__name__] = clr
39-
40-
41-
_load_clr()
5+
from pythonnet import load
6+
load()

‎pythonnet.sln

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Python.Runtime", "src\runti
66
EndProject
77
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Console", "src\console\Console.csproj", "{E6B01706-00BA-4144-9029-186AC42FBE9A}"
88
EndProject
9-
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "clrmodule", "src\clrmodule\clrmodule.csproj", "{F9F5FA13-BC52-4C0B-BC1C-FE3C0B8FCCDD}"
10-
EndProject
119
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Python.EmbeddingTest", "src\embed_tests\Python.EmbeddingTest.csproj", "{819E089B-4770-400E-93C6-4F7A35F0EA12}"
1210
EndProject
1311
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Python.Test", "src\testing\Python.Test.csproj", "{14EF9518-5BB7-4F83-8686-015BD2CC788E}"

‎pythonnet/__init__.py

Lines changed: 71 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,71 @@
1-
def get_assembly_path():
2-
import os
3-
return os.path.dirname(__file__) + "/runtime/Python.Runtime.dll"
1+
import os
2+
import sys
3+
import clr_loader
4+
5+
_RUNTIME = None
6+
_LOADER_ASSEMBLY = None
7+
_FFI = None
8+
_LOADED = False
9+
10+
11+
def set_runtime(runtime):
12+
global _RUNTIME
13+
if _LOADED:
14+
raise RuntimeError("The runtime {runtime} has already been loaded".format(_RUNTIME))
15+
16+
_RUNTIME = runtime
17+
18+
19+
def set_default_runtime() -> None:
20+
if sys.platform == 'win32':
21+
set_runtime(clr_loader.get_netfx())
22+
else:
23+
set_runtime(clr_loader.get_mono())
24+
25+
26+
def load():
27+
global _FFI, _LOADED, _LOADER_ASSEMBLY
28+
29+
if _LOADED:
30+
return
31+
32+
from .find_libpython import linked_libpython
33+
from os.path import join, dirname
34+
35+
if _RUNTIME is None:
36+
# TODO: Warn, in the future the runtime must be set explicitly, either
37+
# as a config/env variable or via set_runtime
38+
set_default_runtime()
39+
40+
dll_path = join(dirname(__file__), "runtime", "Python.Runtime.dll")
41+
libpython = linked_libpython()
42+
43+
if libpython and _FFI is None and sys.platform != "win32":
44+
# Load and leak libpython handle s.t. the .NET runtime doesn't dlcloses
45+
# it
46+
import posix
47+
48+
import cffi
49+
_FFI = cffi.FFI()
50+
_FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL)
51+
52+
_LOADER_ASSEMBLY = _RUNTIME.get_assembly(dll_path)
53+
54+
func = _LOADER_ASSEMBLY["Python.Runtime.Loader.Initialize"]
55+
if func(f"{libpython or ''}".encode("utf8")) != 0:
56+
raise RuntimeError("Failed to initialize Python.Runtime.dll")
57+
58+
import atexit
59+
atexit.register(unload)
60+
61+
62+
def unload():
63+
global _RUNTIME
64+
if _LOADER_ASSEMBLY is not None:
65+
func = _LOADER_ASSEMBLY["Python.Runtime.Loader.Shutdown"]
66+
if func(b"") != 0:
67+
raise RuntimeError("Failed to call Python.NET shutdown")
68+
69+
if _RUNTIME is not None:
70+
# TODO: Add explicit `close` to clr_loader
71+
_RUNTIME = None

0 commit comments

Comments
 (0)