Skip to content

Implement CLR loading in pure Python #857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
filmor opened this issue Apr 29, 2019 · 26 comments
Closed

Implement CLR loading in pure Python #857

filmor opened this issue Apr 29, 2019 · 26 comments
Milestone

Comments

@filmor
Copy link
Member

filmor commented Apr 29, 2019

To simplify the build process (immensely), I'd like to move the CLR loading for Mono and CoreCLR to Python. I have a working prototype at https://gist.github.com/filmor/cf8a02aefff58be09a57ad2eca3cc246 for CoreCLR using CFFI and will work on an equivalent Mono implementation for this as well. For .NET Framework, I don't know of any public API to embed it into the currently running process, so we'll keep the existing loader for this one.

This implements the first section of the Plan(tm) (https://github.com/pythonnet/pythonnet/wiki/Plan#clr-loading).

@filmor filmor added this to the 2.5.0 milestone Apr 29, 2019
@spinwards
Copy link

I tried the loader from your gist in #857 and it caused a hard crash on windows nano server w/ .net core 2.2 and miniconda.

Originally posted by @spinwards in #612 (comment)

@spinwards
Copy link

spinwards commented May 8, 2019

I've tested it using docker with the mcr.microsoft.com/dotnet/core/sdk:2.2 image with both miniconda and a vanilla python install and a copy of Python.Runtime.dll that targets .net standard 2. In all cases when I call func(), python immediately crashes with a generic "Windows fatal exception: access violation".

I had to modify the build somewhat to get it to run in VS2019, so it might be a problem on my end.

@filmor
Copy link
Member Author

filmor commented May 8, 2019

I don't get this error, it works for me on both Linux and Windows up to the point that it complains about System.Reflection.Emit from Python.NET's CodeGenerator class. I'll update the gist to the latest state.

@spinwards
Copy link

Sure, with the net40 target platform, but I am running this on nanoserver with .net core2.

I need the ReflectionPolyfill class to adapt the DefineDynamicAssembly method for .net standard. If I remove the net40 target framework and build the resulting assembly gives me an access violation.

@filmor
Copy link
Member Author

filmor commented Oct 27, 2019

Work is ongoing in https://github.com/filmor/clr-loader.

I'm able to load up .NET Core (>=3 due to hostfxr changes) and get a callable object in Python for something that has the signature int Func(IntPtr data, int size). I'll implement the same interface for Mono now and incorporate the clrmodule hack from Python.NET.

@filmor
Copy link
Member Author

filmor commented Oct 29, 2019

We are getting there, Mono is implemented as well now, I'll try to add .NET framework support tonight.

@filmor filmor pinned this issue Oct 31, 2019
@filmor
Copy link
Member Author

filmor commented Oct 31, 2019

.NET framework support as well as a simple interface are also implemented now. I will create a PR that drops clrmodule in favour of a pure-python clr.py that uses clr_loader and calls a function with the correct signature (int32 (void*, int32)) in Python.NET to run the initialisation.

@kpoman
Copy link

kpoman commented Dec 5, 2019

Hello, I'm interested on the status of this ! Is there some way to use a dotnet core DLL with pythonnet right now ? And on Linux (both x86 and arm), how do we proceed, as we could install mono and/or MS dotnet core sdk: which one is needed ? How to import, AddReferences, use the assemblies ?
Right now I only errors such as the one below:

System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly.GetExportedTypes(RuntimeAssembly assembly, ObjectHandleOnStack retTypes)
   at System.Reflection.RuntimeAssembly.GetExportedTypes()
   at System.Reflection.Assembly.get_ExportedTypes()

@bneumann
Copy link

Hi, I also tried loading a .NET Core 2.0, 2.1 or 3.1 dll using pythonnet. It failed just like @kpoman 's attempt. So I dug a little deeper and it seems, without knowing any details, that pythonnet is loading "some" Runtime when you call import clr in my case it adds C:\Windows\Microsoft.NET\Framework64\v4.0.30319

So @filmor: Is there a way I can tell pythonnet to use another directory? Because I can build (rather publish) my dotnetcore application as self contained. So in my mind it would make sense to tell pythonnet to just use this directory where I have my application DLLs + the Runtime to load all of it.

Here some print out where it shows that the System.Runtime 4.0.0.0 is loaded from my default installation and the attempt to load a different Runtime afterwards fails (of course)

System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
demo, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()
## Could not load file or assembly 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' or one of its dependencies. An attempt was made to load a program with an incorrect format.
System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()

@filmor
Copy link
Member Author

filmor commented Dec 18, 2019

This is much more complicated than you make it out to be :). The reason this works in the way it currently does is because of some magic with regard to how Windows deals with .NET DLLs. There is startup code in there that loads mscorlib that starts up the .NET (Framework) runtime. As far as I can tell you can't use the same mechanism to load .NET Core. You can definitely not use it on Linux or OS X where the runtime linker deals with ELF or Mach-O format libraries, respectively.

I have already started work on this, but don't expect anything usable before January. The loader as such already works (see https://github.com/filmor/clr-loader) for all dynamic .NET runtimes (Framework, Core, Mono), now I'm in the process of refactoring Python.NET to make use of this in #1015.

@bneumann
Copy link

Yeah I thought it might be. I'll check out your loader and look around. Maybe I can help out with something (doubtful but I'll try)

@mungojam
Copy link

Appreciate all your hard work and brains on this 🙂. What's the answer to the question about self-contained exe/dlls, will those be loadable under the new mechanism such that users won't need .net core 3 installed? Happy to raise a separate issue about it if it needs something different doing

@filmor
Copy link
Member Author

filmor commented Dec 18, 2019

@mungojam Contributions to clr-loader on this topic are welcome, I didn't look into this yet. Essentially, all you need to do is locating the hostfxr library, this is currently done in the constructor of HostFxr in https://github.com/filmor/clr-loader/blob/master/clr_loader/hostfxr.py.

@mungojam
Copy link

@filmor thanks for the pointer. I've just taken a look at the output from a simple self-contained build and it contains hostfxr.dll:

image

I reckon the easy way then with the existing code will be to set the DOTNET_ROOT environment variable within the python process and point it at the self-contained folder. I guess if there is more than one self-contained library being consumed, then it just needs to be told about one of them as every hostfxr.dll should be the same.

I will give this a spin at some point to fully confirm.

The only way I can see it could work automatically would be if the hostfxr loader would wait until at least one DLL was referenced so that it could go looking into the same folder as those DLLs, but that sounds complicated and not worth it.

@lostmsu
Copy link
Member

lostmsu commented Mar 3, 2020

@filmor , given this will likely require #1015 to ever work, should it be moved to 3.x?

@filmor
Copy link
Member Author

filmor commented Mar 4, 2020

@lostmsu We can just rename 2.5 to 3.0.

@lostmsu
Copy link
Member

lostmsu commented Mar 4, 2020

There's already 3.0 :-D

@pythonnet pythonnet deleted a comment from hamzashoukat94 Mar 24, 2020
@lostmsu lostmsu modified the milestones: 2.5.0, 3.0.0 Apr 23, 2020
@hillin
Copy link

hillin commented Jun 2, 2020

A mild bump, what's the state of this issue? When would it be possible to load a .net core assembly?

@filmor
Copy link
Member Author

filmor commented Jun 9, 2020

@hillin Development happens completely in the open. If you don't see any progress on this issue in form of comments or commits, then there is no progress.

@pythonnet pythonnet deleted a comment from hamzashoukat94 Jun 10, 2020
@lostmsu
Copy link
Member

lostmsu commented Oct 30, 2020

@filmor have you been working on this lately? What is the status?

@lostmsu
Copy link
Member

lostmsu commented Nov 20, 2020

I've given this some thought, and now think this should ideally be a separate Python module to be published for pip, that the main repository should depend on.

e.g. pip install hostfxr followed by

import hostfxr

hostfxr.init()
hostfxr.get_delegate()

BTW, the linked notebook uses the "old" way to load CLR. The "new" way is through hostfxr + optional nethost to locate hostfxr.

@filmor
Copy link
Member Author

filmor commented Nov 20, 2020

That's pretty much exaxtly what https://github.com/filmor/clr-loader does, with the added advantage that it also implements loaders for Mono and .Net Framework. It also uses hostfxr instead of the old style and works with .NET 5.

@lostmsu
Copy link
Member

lostmsu commented Nov 20, 2020

Do you want to migrate it under pythonnet (as a separate repo) and publish a package? Does it need polishing?

@filmor
Copy link
Member Author

filmor commented Nov 20, 2020

All of those things, yes.

I'll move it under the project now. There are plenty of TODOs, in particular CI and testing. We can publish a package once it's actually used in pythonnet.

@lostmsu
Copy link
Member

lostmsu commented Nov 20, 2020

I think it makes sense to publish package right away as 0.x "alpha", then add CI + testing.

@filmor
Copy link
Member Author

filmor commented Nov 20, 2020

Done: https://pypi.org/project/clr-loader/0.1.0/

@filmor filmor mentioned this issue Dec 13, 2020
4 tasks
@filmor filmor unpinned this issue Feb 13, 2021
@filmor filmor closed this as completed Feb 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants