Skip to content

Commit 6510ff7

Browse files
filmorfilmor
authored
Merge pull request #1373 from filmor/clr-loader
New loading based on clr_loader
2 parents 4da1954 + c1a01b7 commit 6510ff7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

76 files changed

+988
-943
lines changed

‎.github/workflows/main.yml

Lines changed: 22 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -12,33 +12,9 @@ jobs:
1212
fail-fast: false
1313
matrix:
1414
os: [windows, ubuntu, macos]
15-
pyver_minor: [6, 7, 8, 9]
15+
python: ["3.6", "3.7", "3.8", "3.9"]
1616
platform: [x64]
1717
shutdown_mode: [Normal, Soft]
18-
include:
19-
- os: ubuntu
20-
pyver_minor: 6
21-
dll_suffix: m
22-
- os: ubuntu
23-
pyver_minor: 7
24-
dll_suffix: m
25-
26-
- os: macos
27-
dll_prefix: lib
28-
dll_pyver_major: '3.'
29-
dll_suffix: m
30-
- os: ubuntu
31-
dll_prefix: lib
32-
dll_pyver_major: '3.'
33-
- os: windows
34-
dll_pyver_major: '3'
35-
36-
- os: ubuntu
37-
dll_ext: .so
38-
- os: windows
39-
dll_ext: .dll
40-
- os: macos
41-
dll_ext: .dylib
4218

4319
env:
4420
PYTHONNET_SHUTDOWN_MODE: ${{ matrix.SHUTDOWN_MODE }}
@@ -56,10 +32,10 @@ jobs:
5632
- name: Setup .NET
5733
uses: actions/setup-dotnet@v1
5834

59-
- name: Set up Python 3.${{ matrix.pyver_minor }}
35+
- name: Set up Python ${{ matrix.python }}
6036
uses: actions/setup-python@v2
6137
with:
62-
python-version: 3.${{ matrix.pyver_minor }}
38+
python-version: ${{ matrix.python }}
6339
architecture: ${{ matrix.platform }}
6440

6541
- name: Install dependencies
@@ -68,31 +44,34 @@ jobs:
6844
6945
- name: Build and Install
7046
run: |
71-
python setup.py configure
7247
pip install -v .
7348
74-
# TODO this should be gone once clr module sets PythonDLL or preloads it
75-
- name: Python Tests
76-
run: pytest
77-
if: ${{ matrix.os != 'macos' }}
78-
env:
79-
PYTHONNET_PYDLL: ${{ matrix.DLL_PREFIX }}python${{matrix.DLL_PYVER_MAJOR}}${{matrix.PYVER_MINOR}}${{matrix.DLL_SUFFIX}}${{matrix.DLL_EXT}}
49+
- name: Set Python DLL path (non Windows)
50+
if: ${{ matrix.os != 'windows' }}
51+
run: |
52+
python -m pythonnet.find_libpython --export >> $GITHUB_ENV
8053
81-
- name: Python Tests
82-
run: pytest
83-
if: ${{ matrix.os == 'macos' }}
54+
- name: Set Python DLL path (Windows)
55+
if: ${{ matrix.os == 'windows' }}
56+
run: |
57+
python -m pythonnet.find_libpython --export | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
58+
59+
- name: Python Tests (Mono)
60+
if: ${{ matrix.os != 'windows' }}
61+
run: pytest --runtime mono
62+
63+
- name: Python Tests (.NET Core)
64+
run: pytest --runtime netcore
65+
66+
- name: Python Tests (.NET Framework)
67+
if: ${{ matrix.os == 'windows' }}
68+
run: pytest --runtime netfx
8469

8570
- name: Embedding tests
8671
run: dotnet test --runtime any-${{ matrix.platform }} src/embed_tests/
87-
if: ${{ matrix.os != 'macos' }} # Not working right now, doesn't find libpython
88-
env:
89-
PYTHONNET_PYDLL: ${{ matrix.DLL_PREFIX }}python${{matrix.DLL_PYVER_MAJOR}}${{matrix.PYVER_MINOR}}${{matrix.DLL_SUFFIX}}${{matrix.DLL_EXT}}
9072

9173
- name: Python tests run from .NET
9274
run: dotnet test --runtime any-${{ matrix.platform }} src/python_tests_runner/
93-
if: ${{ matrix.os == 'windows' }} # Not working for others right now
94-
env:
95-
PYTHONNET_PYDLL: ${{ matrix.DLL_PREFIX }}python${{matrix.DLL_PYVER_MAJOR}}${{matrix.PYVER_MINOR}}${{matrix.DLL_SUFFIX}}${{matrix.DLL_EXT}}
9675

9776
# TODO: Run perf tests
9877
# TODO: Run mono tests on Windows?

‎.gitignore

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
11
/src/runtime/interopNative.cs
22

3-
# Configuration data
4-
configured.props
5-
63
# General binaries and Build results
74
*.dll
85
*.exe

‎CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@ when .NET expects an integer [#1342][i1342]
3737
- BREAKING: to call Python from .NET `Runtime.PythonDLL` property must be set to Python DLL name
3838
or the DLL must be loaded in advance. This must be done before calling any other Python.NET functions.
3939
- Sign Runtime DLL with a strong name
40+
- Implement loading through `clr_loader` instead of the included `ClrModule`, enables
41+
support for .NET Core
4042

4143
### Fixed
4244

‎Directory.Build.props

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,4 @@
1818
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
1919
</PackageReference>
2020
</ItemGroup>
21-
<Import Project="$(MSBuildThisFileDirectory)configured.props" Condition="Exists('$(MSBuildThisFileDirectory)configured.props')" />
2221
</Project>

‎appveyor.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,9 @@ environment:
3030
init:
3131
# Update Environment Variables based on matrix/platform
3232
- set PY_VER=%PYTHON_VERSION:.=%
33-
- set PYTHONNET_PYDLL=python%PY_VER%.dll
3433
- set PYTHON=C:\PYTHON%PY_VER%
3534
- if %PLATFORM%==x64 (set PYTHON=%PYTHON%-x64)
35+
- set PYTHONNET_PYDLL=%PYTHON%\python%PY_VER%.dll
3636

3737
# Put desired Python version first in PATH
3838
- set PATH=%PYTHON%;%PYTHON%\Scripts;%PATH%
@@ -42,7 +42,6 @@ install:
4242
- pip install --upgrade -r requirements.txt --quiet
4343

4444
build_script:
45-
- python setup.py configure
4645
# Create clean `sdist`. Only used for releases
4746
- python setup.py --quiet sdist
4847
- python setup.py bdist_wheel

‎clr.py

Lines changed: 2 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -2,40 +2,5 @@
22
Legacy Python.NET loader for backwards compatibility
33
"""
44

5-
def _get_netfx_path():
6-
import os, sys
7-
8-
if sys.maxsize > 2 ** 32:
9-
arch = "amd64"
10-
else:
11-
arch = "x86"
12-
13-
return os.path.join(os.path.dirname(__file__), "pythonnet", "netfx", arch, "clr.pyd")
14-
15-
16-
def _get_mono_path():
17-
import os, glob
18-
19-
paths = glob.glob(os.path.join(os.path.dirname(__file__), "pythonnet", "mono", "clr.*so"))
20-
return paths[0]
21-
22-
23-
def _load_clr():
24-
import sys
25-
from importlib import util
26-
27-
if sys.platform == "win32":
28-
path = _get_netfx_path()
29-
else:
30-
path = _get_mono_path()
31-
32-
del sys.modules[__name__]
33-
34-
spec = util.spec_from_file_location("clr", path)
35-
clr = util.module_from_spec(spec)
36-
spec.loader.exec_module(clr)
37-
38-
sys.modules[__name__] = clr
39-
40-
41-
_load_clr()
5+
from pythonnet import load
6+
load()

‎pyproject.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
11
[build-system]
22
requires = ["setuptools>=42", "wheel", "pycparser"]
33
build-backend = "setuptools.build_meta"
4+
5+
[tool.pytest.ini_options]
6+
xfail_strict = true
7+
testpaths = [
8+
"tests",
9+
]

‎pythonnet.sln

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Python.Runtime", "src\runti
66
EndProject
77
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Console", "src\console\Console.csproj", "{E6B01706-00BA-4144-9029-186AC42FBE9A}"
88
EndProject
9-
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "clrmodule", "src\clrmodule\clrmodule.csproj", "{F9F5FA13-BC52-4C0B-BC1C-FE3C0B8FCCDD}"
10-
EndProject
119
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Python.EmbeddingTest", "src\embed_tests\Python.EmbeddingTest.csproj", "{819E089B-4770-400E-93C6-4F7A35F0EA12}"
1210
EndProject
1311
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Python.Test", "src\testing\Python.Test.csproj", "{14EF9518-5BB7-4F83-8686-015BD2CC788E}"
@@ -55,7 +53,6 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Python.PythonTestsRunner",
5553
EndProject
5654
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Build", "Build", "{142A6752-C2C2-4F95-B982-193418001B65}"
5755
ProjectSection(SolutionItems) = preProject
58-
configured.props = configured.props
5956
Directory.Build.props = Directory.Build.props
6057
EndProjectSection
6158
EndProject

‎pythonnet/__init__.py

Lines changed: 71 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,71 @@
1-
def get_assembly_path():
2-
import os
3-
return os.path.dirname(__file__) + "/runtime/Python.Runtime.dll"
1+
import os
2+
import sys
3+
import clr_loader
4+
5+
_RUNTIME = None
6+
_LOADER_ASSEMBLY = None
7+
_FFI = None
8+
_LOADED = False
9+
10+
11+
def set_runtime(runtime):
12+
global _RUNTIME
13+
if _LOADED:
14+
raise RuntimeError("The runtime {runtime} has already been loaded".format(_RUNTIME))
15+
16+
_RUNTIME = runtime
17+
18+
19+
def set_default_runtime() -> None:
20+
if sys.platform == 'win32':
21+
set_runtime(clr_loader.get_netfx())
22+
else:
23+
set_runtime(clr_loader.get_mono())
24+
25+
26+
def load():
27+
global _FFI, _LOADED, _LOADER_ASSEMBLY
28+
29+
if _LOADED:
30+
return
31+
32+
from .find_libpython import linked_libpython
33+
from os.path import join, dirname
34+
35+
if _RUNTIME is None:
36+
# TODO: Warn, in the future the runtime must be set explicitly, either
37+
# as a config/env variable or via set_runtime
38+
set_default_runtime()
39+
40+
dll_path = join(dirname(__file__), "runtime", "Python.Runtime.dll")
41+
libpython = linked_libpython()
42+
43+
if libpython and _FFI is None and sys.platform != "win32":
44+
# Load and leak libpython handle s.t. the .NET runtime doesn't dlcloses
45+
# it
46+
import posix
47+
48+
import cffi
49+
_FFI = cffi.FFI()
50+
_FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL)
51+
52+
_LOADER_ASSEMBLY = _RUNTIME.get_assembly(dll_path)
53+
54+
func = _LOADER_ASSEMBLY["Python.Runtime.Loader.Initialize"]
55+
if func(f"{libpython or ''}".encode("utf8")) != 0:
56+
raise RuntimeError("Failed to initialize Python.Runtime.dll")
57+
58+
import atexit
59+
atexit.register(unload)
60+
61+
62+
def unload():
63+
global _RUNTIME
64+
if _LOADER_ASSEMBLY is not None:
65+
func = _LOADER_ASSEMBLY["Python.Runtime.Loader.Shutdown"]
66+
if func(b"") != 0:
67+
raise RuntimeError("Failed to call Python.NET shutdown")
68+
69+
if _RUNTIME is not None:
70+
# TODO: Add explicit `close` to clr_loader
71+
_RUNTIME = None

0 commit comments

Comments
 (0)