-
Notifications
You must be signed in to change notification settings - Fork 748
Implicit List conversion breaks backwards compatibility + numpy support #514
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Is this the issue you are experiencing? |
As far as I can tell, this isn't like #451 because the error in #451 occurs in .NET. The error in this issue occurs in Python. But, the same underlying problem may be causing both issues. Given the .NET code above, here is a different demonstration of the poor behavior:
That assertion used to (Computer 1, version 2.3) succeed. Now (Computer 2, ce14424+ on master) it fails. From #451, I see you commented, "The reason is because before the list is passed back to C#, it gets converted to Python list." It seems like this is a new (since 2.3) behavior, right? I suggest that this breaks backward compatibility and is generally undesirable. An explicit helper function to easily convert a generic .NET list to a Python list might be nice, but the conversion should not be performed automatically. But if it is, nonetheless, performed automatically, then the symmetric operation (automatically converting a Python list to a generic .NET list) MUST also happen. Also from #451: "But this is of course a lot of conversion back and forth and maybe a performance issue." I agree, which is why I strongly suggest not performing automatic conversion. EDIT: The reason I say a helper function to convert a .NET List to a Python list only might be nice is that the conversion is trivial: |
@BenjaminPelletier I agree that converting the CLR At the same time we need to get this sample with numpy arrays working seamlessly: https://github.com/pythonnet/pythonnet#example This is why the conversion was introduced. |
Calling Python from .NET is outside my use case so I'm not familiar with the logistics doing so. But "just like I'd expect" would be for pythonnet to expose explicit .NET types that map to Python types for the user to instantiate (in .NET) by some means before calling Python with them. These types would probably have helper functions for converting/wrapping from common corresponding .NET types. FWIW, here's how I'm marshalling arrays between .NET and NumPy, and it seems like it would have been nice to not have to figure out & write this myself: Python:
.NET:
|
@BenjaminPelletier if you would like to submit a helper function for numpy arrays, then pull requests are welcome! Here is another example of passing numpy arrays: https://stackoverflow.com/questions/43910749/efficiently-convert-system-single-to-numpy-array |
@denfromufa I also have a numpy array converter here With this converter, the example https://github.com/pythonnet/pythonnet#example can be replaced with this one,
|
@yagweb do you suggest that we should revert back the auto-conversion due to this regression and use your numpy converter? |
which commit broke this ? Was it the one I added to fix the example? |
@vmuriart this pull request broke C# List behavior: |
ok to revert behavior until better solution is found |
Hi, I wrote a Python-side Numpy <-> .NET conversion functions that use
The
I don't see any evidence of memory-leaking. Edit: one can do a zero-copy with |
@robbmcleod I just wanted to say thanks for your code and give feedback - it works great! I especially appreciate not having .NET wrapper and handling all conversion in Python/pythonnet/ctypes. |
I have a use-case that is also broken by the #427 change and doesn't have a good workaround.
Once the List is converted to a python list, any mutations to the python list don't get updated in the original C# class.
This breaks any case where you would have some other C# method on the class that needs to use the list after it's been modified by python. |
Tangentially related, but in our fork I just added a way to extend how .NET types are converted to Python and vice versa (so far without disabling conversions, that are coming out of the box, but I am planning this too). See losttech@c0ffc29 |
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for pythonnet#514
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for pythonnet#514
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for pythonnet#514
The new |
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for #514
This should now be solved by codecs: Codecs: .NET <-> Python conversions |
Thanks. |
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for pythonnet#514
@lostmsu Hi there. Would you be able to provide an example of how to use Codecs to prevent the conversion from a System.Collections.Generic.List to a python list? I read both the Codec wiki page and this example but I didn't get it, sorry. |
Sorry, I don't do that. I think I got associated with #514 because I reported that Pythonnet 2.x had broken support for .NET Remoting, and whoever fixed it (maybe?) introduced an issue related to what you are talking about. So that was holding up the release containing the .NET Remoting fix. Since the fix for me has been released, I'm guessing your issue also got resolved.
--------- Original Message --------- Subject: Re: [pythonnet/pythonnet] Implicit List conversion breaks backwards compatibility + numpy support (#514)
From: "ddsleonardo" <notifications@github.com>
Date: 10/30/20 5:38 am
To: "pythonnet/pythonnet" <pythonnet@noreply.github.com>
Cc: "jcarole" <github@jnpcarole.com>, "Manual" <manual@noreply.github.com>
@lostmsu Hi there. Would you be able to provide an example of how to use Codecs to prevent the conversion from a System.Collections.Generic.List to a python list? I read both the Codec wiki page and this example but I didn't get it, sorry.
Is there a way to just set this behaviour by default anytime I import pythonnet?
-
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
I do not understand your reply @jcarole. |
@ddsleonardo have you checked the new test named |
Sorry to confuse you. I mean to say that I don't know anything about the issue you're referring to and tried to explain why you might have thought I was at all involved in that topic.
…--------- Original Message --------- Subject: Re: [pythonnet/pythonnet] Implicit List conversion breaks backwards compatibility + numpy support (#514)
From: "ddsleonardo" <notifications@github.com>
Date: 10/30/20 9:41 am
To: "pythonnet/pythonnet" <pythonnet@noreply.github.com>
Cc: "jcarole" <github@jnpcarole.com>, "Mention" <mention@noreply.github.com>
I do not understand your reply @jcarole.
To make it clear, I am having the same problem as #1153 and also mentioned here. In both instances there are mentions either of RawProxyEncoder or the Codecs but I could not understand the usage examples. The only working solution I have found is this comment previously made in this discussion. This comment sounded encouraging, it would be really nice if we could make the conversions optional.
-
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
@lostmsu Thanks for the reply. My main question was relative to at what scope should I use the Codecs, i.e. do I need to define and register an instance of the Encoder class every time I open a Python Interactive and want to manipulate some C# Lists? (which I believe is the case, correct)? I think I managed to follow Still following on the same question, it sounds like this has been made a lot more convoluted than maybe it should be. I had been running a previous version of pythonnet for a while to avoid this, and only encountered the issue again as I had to reset my environment. Now I appreciate that this package isn't written just for me, but I'd be interested to know the motivations behind this, as it seems that I'm not the only one who has struggled with this and the workarounds to make it work are not trivial. At the same time, I am happy to run my own fork of this repo without the conversion, and set up a pipeline that keeps everything else up to date with the main repo. |
@ddsleonardo don't worry, the default conversion is going to be off by default in the next major version. I was not here around the time Python.NET started, but I suspect this conversion was added to make most APIs easily callable: |
Updated my code snippet for PythonNet 2.5.1, Python 3.9.4, and NumPy 1.20.2 for those who are looking for a working solution: https://gist.github.com/robbmcleod/73ca42da5984e6d0e5b6ad28bc4a504e |
@robbmcleod I adapted your last code so that complex arrays in python convert back-and-forth with SystemArray in .NET whose element type are directly System.Numerics.Complex (instead of adding extra dimension): https://gist.github.com/CitizenInsane/c8d3ddc5b14faceec433af3e940e22a8 |
@CitizenInsane thanks although in my case I actually do need the 64-bit complex as that's what the counterparty software that I'm talking to uses. |
@robbmcleod Yes it all depends. The .NET API i try to connect with is using System.Numerics.Complex arrays directly. Thanks for the great job, first time I'm doing python and this to interop with .net so it helped a lot to have your converters :). |
BTW, @robbmcleod , @CitizenInsane , in pythonnet 3.0 .NET arrays will implement buffer protocol! |
Good to know, thanks, that should be a big improvement over my memcpy solution. |
@yagweb Hi there, I found that in your converter, "using(Py.GIL())" is called multiple times in a function. However, every time that I call the function on my code for the second time, the code just hangs at "using(Py.GIL())". I have to restart the whole program to get the function to work for the second time. Could you please advise how the system should be set up to solve this problem? Thank you so much! |
Thanks I decided to add missing part of reply in regards to how convert numpy array from c# side. All that left to do is load your script as module and call these functions. Copy script that robbmcleod posted and save as Put this into c# project using Python.Runtime;
public static class NumpyUtils
{
public static string script = File.ReadAllText("array_converter.py");
public static dynamic ArrayConverterModule(){
return PyModule.FromString("",script);
}
} Call it from c# and cast types right away dynamic np_array = ...; //some numpy float32 array
dynamic array_converter = NumpyUtils.ArrayConverterModule();
float[] converted array_converter.asNetArray(np_array); Also I would like to add script that I use for slicing numpy arrays using Python.Runtime;
public static class NumpyUtils
{
public static string script = File.ReadAllText("array_converter.py");
public static dynamic ArrayConverterModule(){
return PyModule.FromString("",script);
}
/// <summary>
/// Slices numpy array
/// </summary>
/// <param name="slice">Slice without object name and brackets []. For example : ":5,3"</param>
public static dynamic Slice(PyObject np_array,string slice)
{
using var glob = new PyDict();
glob["arr"] = np_array;
var sliced = PythonEngine.Eval($"arr[{slice}]",
globals: glob
);
return sliced;
}
} dynamic np_array = ...; //some numpy float32 array
var firstFive = NumpyUtils.Slice(np_array,":5"); |
Environment
Computer 1:
Computer 2:
Details
Description: Python calls .NET function which returns a List. Python then passes the return value, without modification, to a second .NET function which accepts a List. Computer 1 executes this code just fine. On Computer 2, there is no .NET function found that matches the arguments because the return value of the .NET function has been transformed into a Python list.
Python code:
.NET code:
The Python code works fine on Computer 1. On Computer 2, the PrintDoubleList call produces
TypeError: No method matches given arguments
If I print type(x) in Python, Computer 1 gives me a .NET List type while Computer 2 gives me a Python list. I can print x.Count on Computer 1, but I get a missing attribute error on Computer 2.
If I build manually from the 2.3 tag, I get the same (good) behavior as on Computer 1.
It seems that some feature has been partially added to automatically convert .NET objects into Python objects when possible. I suppose this is ok (though I would prefer that not happen because I don't want the mandatory performance hit of converting even when I don't want to convert), but if that's the intention, there must be automatic conversion of Python objects to .NET objects also. One without the other is a critical bug.
The text was updated successfully, but these errors were encountered: