Note that for .NET 4+ still have to use unmanaged exports.

apparently @lstratman has extensive experience porting Mono to CoreCLR embedding.

Thanks @denfromufa :).

It's funny, I actually had a fork of this repo from a while back with the goal of looking into this. But, since my company has only a single customer that might want to integrate our .NET server application into their Python application on Linux and, even then, only at some point in the future, it's down towards the bottom of my priority list for development unfortunately. However, I'm happy to offer some technical guidance.

I implemented basically this same functionality for Node.js for the Edge.js project and a lot of the code that I wrote there can probably be re-used or easily re-purposed for pythonnet. https://github.com/tjanczuk/edge/blob/master/src/CoreCLREmbedding/coreclrembedding.cpp#L192 contains the code that spins up the .NET Core CLR in any process. Towards the end of the method (https://github.com/tjanczuk/edge/blob/master/src/CoreCLREmbedding/coreclrembedding.cpp#L673), we get pointers to a bunch of .NET methods that serve as the managed end of the Node.js->.NET interop process. These function pointers are then invoked anytime that we need to call from Node.js to .NET. Implementations of those .NET methods are in https://github.com/tjanczuk/edge/blob/master/src/double/Edge.js/dotnetcore/coreclrembedding.cs and are included in the .NET Core application that's run from within Edge.js through a NuGet package that needs to be referenced (see https://github.com/tjanczuk/edge#how-to-specify-additional-clr-assembly-references-in-c-code).

This works well for Edge.js because the interaction between Node.js and .NET is all encapsulated in functions in one language or the other. So to call between the two domains, you're always invoking functions, never creating variables, setting properties, etc. However for pythonnet, we're obviously more "integrated" into the Python syntax, so we can do things like create variables, set properties, etc. You can write more methods in the .NET end of the interop process to facilitate this, or try to stay in unmanaged code as much as possible by calling the .NET Core CLR native methods to do things like create variables and set properties. That would be the more performant, but potentially more difficult, route. This will have to be done through the ICLRRuntimeHost instance that you get back from calling coreclr_initialize(). It should theoretically be possible, but is left as an exercise to the interested reader.

Anyway, that concludes my brain dump, I hope that it helps. Feel free to post questions and I'll do my best to answer them. At some point, I may have some time for actual development as well, but can't offer any promises.

@lstratman thank you for very detailed response, there is a question from MS if CoreCLR C-API is good enough, would you mind replying there? MS is considering standardizing C-API for .NET ecosystem.

https://blogs.msdn.microsoft.com/dotnet/2016/09/26/introducing-net-standard/#comment-138395

Are the four methods (Initialize, Shutdown, Create_Delegate, Execute_Assembly), mentioned enough for embedding CoreCLR in pythonnet?

I've replied to that thread that those methods are sufficient. Looking further into the existing pythonnet code, I realized that it does all of the things that I mentioned earlier (create variables, set properties, index arrays, etc.) in managed code anyway, so once the "spinning up CoreCLR" code is integrated from Edge.js, pythonnet should just be able to re-use a lot of its existing codebase.

Another thing holding of CoreCLR port is Reflection.Emit used in pythonnet in one place:

https://github.com/dotnet/corefx/issues/4491#issuecomment-189756092

But we could possibly just do conditional compilation for this feature.

otherwise try ikvm.reflection.emit or mono.cecil

Hi, I started work toward Python.Net Core edition in this fork https://github.com/var1ap/pythonnet , stay tuned.

@var1ap this is great news, do you agree to transition from Zope to MIT license, if your port is successful?

@denfromufa of course agree.

Greate news!
https://github.com/var1ap/pythonnet/commit/2b508fd3fda1f67db6785b03a409b9edd0e67686
I managed to get pythonnet work under dotnet core 1.1(Win x64)

Also pythonnet successfully ported to dotnet core 1.1(Linux x64 Ubuntu 16.04) (in the same fork).
This is just Proof of concept. So somebody should port this project to CoreCLR with good code quality required for production projects.

Hi guys! Pythonnet really works under dotnet core >= 1.0(NetStandard1.5). Thank you very much var1ap for your research. I continued to work on porting pythonnet to dotnet core in this repo https://github.com/dmitriyse/pythonnet/tree/coreclr. Also I plan to improve my binary distribution and avoid any dll load magic. Finally we should get good nuget based distribution.

@var1ap @dmitriyse can you prepare a pull request with compilation flags for coreclr? much like .NET/Mono flags that we have currently.

I can spend time to reapply coreclr code adoption in the next week.
Please review talk in those posts dmitriyse@38adb2e#commitcomment-20774083

Please consider with vmuriart, which fork/branch is most suitable for this work, where I should fork.

@dmitriyse on top of master, but at the rate we are committing changes at the moment it wouldn't make sense to do all the work if it won't get merged in immediately.

@denfromufa, after reviewing @dmitriyse and @var1ap, their changes look pretty straight forward to upgrade to coreclr. My hesitation is that it seems that it would require us to upgrade from 4.0 to 4.5 framework which may drop users (I still use 4.0). We could throw in a bunch of #if directives, but it would make the code very messy. (side note, We actually got rid of most mono_linux and mono_osx flags from the code thanks to @dmitriyse)

Another issue is that coreclr is still undergoing changes wrt their build system. They've gone from project.json back to an upgraded msbuild with new project file formats. mono wouldn't compile against either of these options; ie. we would need to have two sets of csproj/sln/packages.config files in order to ensure mono can still be used for compiling (unless we drop mono?)

Currently what I'm doing is slowly porting over some of the changes @dmitriyse and @var1ap did to minimize the changes for when we migrate to coreclr. I'm hoping to find a clean way to maintain clr 4.0 and add coreclr, but at the moment I'm not finding it and it might need to be part of pythonnet 3.0.

Hi! I restarted work on porting PythonNet to CoreCLR. Now it can be performed as additional files.
So current build and new build (based on .csproj 2017) can coexist side-by-side.
https://github.com/dmitriyse/pythonnet/tree/coreclr

@dmitriyse this looks much cleaner! Can you open a pull request for review cycle?

Ok, I will create pool request right after some useful point. I fixed 102 from 120 migration issues to get pythonnet working with at least one build configuration.

Half of the problem solved - python can be embedded in to CoreCLR 2.0 (#519)

System.Reflection.Emit works good under CoreCLR 2.0 except for CDecl calling convention. CDecl currently unsupported in the CoreCLR 2.0

As a workaround I changed XDecref/XIncref to Py_DecRef/, Py_IncRef.
And used UnamanagedFunctionPoinerAttribute with Marshal.GetDelegateForFunctionPointer, that works good, and does not looks too slow.

The rest part is to inject CoreCLR into python through CoreCLR C Api.

@dmitriyse for cdecl did you mean this issue? https://github.com/dotnet/corefx/issues/9800

Yes, exactly this flaw.

@dmitriyse for rest part to inject CoreCLR into Python we can avoid CoreCLR C API, if the DLLExport becomes stable with .NET Core, @3F has to use custom CoreCLR at the moment. DLLExport is the next generation of UnmanagedExports, that we are currently using in .NET Framework:

https://github.com/3F/DllExport

@dmitriyse with your valid finalizer from #532 I'm getting this unexpected crash on .NET Core 2.0, which does not appear on .NET Framework:

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7\netcoreapp2.0>dotnet nPython.dll
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> import System
>>> System.Double(1.0)

Unhandled Exception: System.InvalidCastException: Unable to cast object of type 'TernaryFunc' to type 'IntPtr_3_Delegate'.
   at Python.Runtime.NativeCall.Call_3(IntPtr fp, IntPtr a1, IntPtr a2, IntPtr a3)
   at Python.Runtime.MetaType.tp_call(IntPtr tp, IntPtr args, IntPtr kw)
   at Python.Runtime.Runtime.Py_Main(Int32 argc, String[] argv)
   at Python.Runtime.PythonConsole.Main(String[] args)

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7\netcoreapp2.0>cd..

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7>nPython
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> import System
>>> System.Double(1.0)
<System.Double object at 0x052510B0>

@djoyce82 this is great! definitely, please submit a pull request.

@djoyce82 That's great work, does it build and run from Linux? What commands did you use to build it and run the tests?

@djoyce82

python setup.py bdist_wheel --xplat

Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Python.Runtime.15 -> /users/isys/icronyn/git/pythonnet_clr/src/runtime/bin/netstandard2.0/Python.Runtime.dll
  Console.15 -> /users/isys/icronyn/git/pythonnet_clr/src/console/bin/netcoreapp2.0/nPython.dll
  Console.15 -> /users/isys/icronyn/git/pythonnet_clr/build/lib.linux-x86_64-2.7/netcoreapp2.0/
  Python.EmbeddingTest.15 -> /users/isys/icronyn/git/pythonnet_clr/src/embed_tests/bin/netcoreapp2.0/Python.EmbeddingTest.dll
  Python.EmbeddingTest.15 -> /users/isys/icronyn/git/pythonnet_clr/src/embed_tests/bin/netcoreapp2.0_publish/
building 'clr' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
creating build/temp.linux-x86_64-2.7/src/coreclr
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/man/releases/python-medusa/27-1/include/python2.7 -c src/coreclr/pynetinit.c -o build/temp.linux-x86_64-2.7/src/coreclr/pynetinit.o
src/coreclr/pynetinit.c: In function \u2018init\u2019:
src/coreclr/pynetinit.c:112:5: error: \u2018for\u2019 loop initial declarations are only allowed in C99 mode
     for (int ii = 0; ii < PyList_Size(syspath); ++ii)

@Cronan pass '-std=c99' to compile args:

https://stackoverflow.com/a/10867041/2230844

Thanks @denfromufa, I fixed it by declaring the variable outside the loop - there was a single other case of this:

src/coreclr/coreutils.c: In function \u2018AddFilesFromDirectoryToTpaList\u2019:
src/coreclr/coreutils.c:236:5: error: \u2018for\u2019 loop initial declarations are only allowed in C99 mode
     for (size_t extIndex = 0; extIndex < sizeof(tpaExtensions) / sizeof(tpaExtensions[0]); extIndex++)
     ^
src/coreclr/coreutils.c:236:5: note: use option -std=c99 or -std=gnu99 to compile your code
error: command 'gcc' failed with exit status 1

@denfromufa @djoyce82 Lots of the tests failed, with this kind of thing:

Unhandled Exception: System.MissingMethodException: Constructor on type 'System.Reflection.Emit.TypeBuilder' not found.
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, Object[] args)
   at Python.Runtime.DelegateManager.GetDelegate(Type dtype, IntPtr callable)
   at Python.Runtime.EventObject.AddEventHandler(IntPtr target, IntPtr handler)
   at Python.Runtime.EventBinding.nb_inplace_add(IntPtr ob, IntPtr arg)
Aborted

@Cronan I believe the test failures are the same issue as detailed in #590 and #571, looks like the implementation of System.Reflection.Emit.TypeBuilder has changed in .NET Core?

PR is #612

If I want to use a .net core assembly in python and only care about using on windows, is that possible with the current python.net version? Thanks!

@weixu02 this is possible via npython.exe if you compile the full pythonnet solution

Let me document the current state of .net core support for the community:

Embedding means calling into Python from .NET (Python.Runtime.DLL)
Extending means calling into .NET from Python (clr.pyd and then Python.Runtime.DLL or npython.exe)

1st "+|-|*" sign means supporting code is available
2nd "+|-|*" sign means tested in CI

"+" means available
"-" means not available
"*" means available via npython.exe

Outstanding issues and PRs in the order of priority:

#612

#531
#613
#590
#565

Hi, I'm trying to test calling from dotnet core into CPython.
If I understand @denfromufa correctly, that should be working.
But how do I set it up? pip install pythonnet fails with the following message:

Could not load file or assembly 'System.ComponentModel.Composition, Version=4.0.0.0,Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.

Installing the source version with python setup.py build_ext --inplace or just executing mono tools/nuget/nuget.exe leads to the same error. My interpretation attempt: mono on my Linuxmint 18.2 system could be too old.

I also tried installing via nuget but got (for framework netcoreapp2.0)

error: Package 'pythonnet' is incompatible with 'all' frameworks in project '/home/matt/dvl/experiments/pythonnet-core/pythonnet-core.csproj'.

Questions:

• Is upgrading mono through their apt repository the way to go?
• Does the status table from @denfromufa maybe reflect the current master but not what is currently released?
• If I want to write dotnet around pythonnet do I still install it via pip not via nuget?

@MatthiasKauer current dotnet support is listed for master branch, mono should be installed according to official mono installation guide (see also .travis.yml), pip is primary channel for full installation, nuget support is limited right now, e.g. see #165.

Sorry for my ignorance. I'm trying to get embedding to work. I've installed pythonnet from nuget (2.3.0-py27-dotnet). But this Python.Runtime.dll doesn't seem to be .Net Standard or -Core. Tried to compile the source code \pythonnet-master\src\runtime\Python.Runtime.csproj to Standard or Core but it seem to be a .Net 4.x. Also tried the Python.Runtime.dll in the Wheel package from here., same problem.
Can someone point me in a direction on how to get my hands on a .Net Standard or -Core version of Python.Runtime.dll?
I read a lot of places that the master branch support .Core....but I really don't know what to do with this information beside what I already tried?

@MrM40 .net core builds are in this solution:

https://github.com/pythonnet/pythonnet/blob/master/pythonnet.15.sln

But I recommend building from python side by passing the --xplat flag to setup.py:

https://github.com/pythonnet/pythonnet/blob/master/appveyor.yml#L60

You can also grab the builds/wheels from appveyor artifacts.

It is not possible to keep all 3 major runtimes (.NET Framework, .NET Core, and Mono) in the same solution/project.

Still running against a wall. I only see release/debug to Mono and Win in those projects?
And I assume I need to compile the "Console.15" project, right
By the way, why not target .NET Standard in these projects?

@MrM40 see this: https://github.com/pythonnet/pythonnet/blob/master/src/runtime/Python.Runtime.15.csproj#L4

Thanks :-)
I'm running on Linux(Debian) and now I get this errir:

Start Unhandled Exception: System.DllNotFoundException: Unable to load DLL 'python27': The specified module or one of its dependencies could not be found. (Exception from HRESULT: 0x8007007E) at Python.Runtime.Runtime.Py_IsInitialized() at Python.Runtime.Runtime.Initialize(Boolean initSigs) in D:\Development\~.Net libraries (3.part)\phytonnet\2018-12-09\pythonnet-master\src\runtime\runtime.cs:line 274 at Python.Runtime.PythonEngine.Initialize(IEnumerable1 args, Boolean setSysArgv, Boolean initSigs) in \pythonnet-master\src\runtime\pythonengine.cs:line 169 at Python.Runtime.Py.GIL() in \pythonnet-master\src\runtime\pythonengine.cs:line 619 at pythonnet_w.Program.Main(String[] args) in \pythonnet_test\Program.cs:line 9 Aborted

I made two links: python2.7 and python27 in the same folder as Python.Runtime.dll pointing to "/usr/bin/python2.7".

Not I got this error:

Start Unhandled Exception: System.EntryPointNotFoundException: Unable to find an entry point named 'PyUnicodeUCS2_FromUnicode' in DLL 'python27'. at Python.Runtime.Runtime.PyUnicode_FromUnicode(String s, IntPtr size) at Python.Runtime.Runtime.Initialize(Boolean initSigs) in \pythonnet-master\src\runtime\runtime.cs:line 337 at Python.Runtime.PythonEngine.Initialize(IEnumerable 1 args, Boolean setSysArgv, Boolean initSigs) in pythonnet-master\src\runtime\pythonengine.cs:line 169 at Python.Runtime.Py.GIL() in \pythonnet-master\src\runtime\pythonengine.cs:line 619 at pythonnet_w.Program.Main(String[] args) in \pythonnet_test\Program.cs:line 9 Aborted

@MrM40 please don't sidetrack this .NET Core (CoreCLR) status with other issues. In a separate issue, document how you are building and how you are using pythonnet to make a reproducible case. Also search the issue tracker for duplicate issues before posting.

I built https://github.com/pythonnet/pythonnet/blob/master/pythonnet.15.sln with the following compilation symbols:

XPLAT;;PYTHON3;PYTHON37;UCS4;NETSTANDARD;

Running this in a .NET Core application on Windows 8.1 causes the program to fatally crash:

The program '[11180] dotnet.exe' has exited with code -1073740791 (0xc0000409).

Without any more information. I'm sure more information can be obtained, but I'm unsure how.

Kind regards, Emil

Please try UCS2

@dmitriyse Thanks! Changed to UCS2, now getting:

The program '[12304] dotnet.exe' has exited with code 255 (0xff).

Unfortunately, I haven't tested this solution widely. Win 8.1 setup hasn't been tested. So you need to dive deeply with VS debugging.

@dmitriyse Hah! I got it working :) I had this line hanging around:

PythonEngine.PythonPath = @"C:\Users\E\AppData\Local\Programs\Python\Python37";

Removing it fixed the issue. Even though I think the path is correct.

Closed in favour of #984.