Skip to content

Implicit List conversion breaks backwards compatibility + numpy support #514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BenjaminPelletier opened this issue Jul 20, 2017 · 37 comments
Labels
Milestone

Comments

@BenjaminPelletier
Copy link

Environment

Computer 1:

  • Pythonnet version: 2.3, installed via pip I think
  • Python version: 2.7.12
  • Operating System: Ubuntu 16.04.2

Computer 2:

  • Pythonnet version: manual build on master from commit ce14424 (currently 11 commits behind)
  • Python version: 2.7.12
  • Operating System: Ubuntu 16.04.2

Details

Description: Python calls .NET function which returns a List. Python then passes the return value, without modification, to a second .NET function which accepts a List. Computer 1 executes this code just fine. On Computer 2, there is no .NET function found that matches the arguments because the return value of the .NET function has been transformed into a Python list.

Python code:

import clr
from MyDotNetProject import PythonInterop

x = PythonInterop.GetDoubleList()
PythonInterop.PrintDoubleList(x)

.NET code:

    public class PythonInterop
    {
        public static List<Double> GetDoubleList() {
            var result = new List<Double>();
            result.Add(1);
            result.Add(2);
            result.Add(3);
            return result;
        }

        public static void PrintDoubleList(List<Double> list) {
            Console.WriteLine("You list has " + list.Count + " elements");
        }
    }

The Python code works fine on Computer 1. On Computer 2, the PrintDoubleList call produces
TypeError: No method matches given arguments

If I print type(x) in Python, Computer 1 gives me a .NET List type while Computer 2 gives me a Python list. I can print x.Count on Computer 1, but I get a missing attribute error on Computer 2.

If I build manually from the 2.3 tag, I get the same (good) behavior as on Computer 1.

It seems that some feature has been partially added to automatically convert .NET objects into Python objects when possible. I suppose this is ok (though I would prefer that not happen because I don't want the mandatory performance hit of converting even when I don't want to convert), but if that's the intention, there must be automatic conversion of Python objects to .NET objects also. One without the other is a critical bug.

@den-run-ai
Copy link
Contributor

den-run-ai commented Jul 20, 2017

Is this the issue you are experiencing?

#451

@BenjaminPelletier
Copy link
Author

BenjaminPelletier commented Jul 20, 2017

As far as I can tell, this isn't like #451 because the error in #451 occurs in .NET. The error in this issue occurs in Python. But, the same underlying problem may be causing both issues. Given the .NET code above, here is a different demonstration of the poor behavior:

import clr
from MyDotNetProject import PythonInterop
from System.Collections.Generic import List
from System import Double

x = PythonInterop.GetDoubleList()
assert isinstance(x, List[Double])

That assertion used to (Computer 1, version 2.3) succeed. Now (Computer 2, ce14424+ on master) it fails.

From #451, I see you commented, "The reason is because before the list is passed back to C#, it gets converted to Python list." It seems like this is a new (since 2.3) behavior, right? I suggest that this breaks backward compatibility and is generally undesirable. An explicit helper function to easily convert a generic .NET list to a Python list might be nice, but the conversion should not be performed automatically. But if it is, nonetheless, performed automatically, then the symmetric operation (automatically converting a Python list to a generic .NET list) MUST also happen.

Also from #451: "But this is of course a lot of conversion back and forth and maybe a performance issue." I agree, which is why I strongly suggest not performing automatic conversion.

EDIT: The reason I say a helper function to convert a .NET List to a Python list only might be nice is that the conversion is trivial:
y = list([x[i] for i in range(x.Count)])

@den-run-ai
Copy link
Contributor

@BenjaminPelletier I agree that converting the CLR List by default is unexpected behavior and should be considered a regression. What do you think @vmuriart ?

At the same time we need to get this sample with numpy arrays working seamlessly:

https://github.com/pythonnet/pythonnet#example

This is why the conversion was introduced.

@BenjaminPelletier
Copy link
Author

BenjaminPelletier commented Jul 20, 2017

Calling Python from .NET is outside my use case so I'm not familiar with the logistics doing so. But "just like I'd expect" would be for pythonnet to expose explicit .NET types that map to Python types for the user to instantiate (in .NET) by some means before calling Python with them. These types would probably have helper functions for converting/wrapping from common corresponding .NET types.

FWIW, here's how I'm marshalling arrays between .NET and NumPy, and it seems like it would have been nice to not have to figure out & write this myself:

Python:

######################## Copy operations #################################

# [Python.NET] Efficient copy of .NET Array to ctypes or numpy array.
# https://mail.python.org/pipermail/pythondotnet/2014-May/001525.html

def copyNumpyToDotNetArray(srcnparray, destdotnetarray):
    ''' Copies the content in a numpy array of any dimensions with dtype=float to a one-dimensional .NET array
    The number of elements in srcnpfloat must not exceed the size of destdotnetdouble
    
    @srcnpfloat: numpy array (any number of dimensions) with dtype=float
    @destdotnetdouble: Pre-allocated .NET Double Array object (via Array.CreateInstance(Double, n) or similar) to copy the numpy array content into
    '''
    if len(srcnparray.shape) == 1 and not srcnparray.dtype == bool:
        ptr = IntPtr.__overloads__[long](srcnparray.__array_interface__['data'][0])
        Marshal.Copy(ptr, destdotnetarray, 0, srcnparray.size)
    else:
        if not srcnparray.flags.c_contiguous:
            srcnparray = srcnparray.copy(order='C')
        assert srcnparray.flags.c_contiguous
        PythonInterop.CopyFromPointer(IntPtr.__overloads__[long](srcnparray.__array_interface__['data'][0]), destdotnetarray)

def copyDotNetArrayToNumpy(srcdotnetarray, destnparray):
    ''' Copies the content of a one-dimensional .NET array into a numpy array of any dimensions
    The number of elements in srcdotnetdouble and destnpfloat must match
    
    @srcdotnetdouble: One-dimensional .NET Array of Doubles (via Array.CreateInstance(Double, n) or similar) containing source data
    @destnpfloat: numpy array (any number of dimensions) to copy the source data into
    
    Use dest = np.empty(desiredShape, dtype=float) to preinitialize if creating a new variable
    '''
    if len(srcdotnetarray) == 1:
        ptr = IntPtr.__overloads__[long](destnparray.__array_interface__['data'][0])
        Marshal.Copy(srcdotnetarray, 0, ptr, srcdotnetarray.Length)
    else:
        PythonInterop.CopyToPointer(srcdotnetarray, IntPtr.__overloads__[long](destnparray.__array_interface__['data'][0]))

######################## Conversions to .NET #############################

def dotnetArrayOf(nparray):
    ''' Creates and returns a .NET Array that mirrors the provided numpy array
    
    @nparray: numpy array
    
    @Returns: .NET Array with element type matching nparray.dtype and identical dimensions with content that matches the provided numpy array
    '''
    dims = nparray.shape
    n = len(dims)
    dims_dn = Array.CreateInstance(Int32, n)
    for i in range(n):
        dims_dn[i] = Int32(dims[i])
    if nparray.dtype == float:
        dn = Array.CreateInstance(Double, dims_dn)
    elif nparray.dtype == int: #NOTE: this check is probably invalid in Python 3.x
        dn = Array.CreateInstance(Int64, dims_dn)
    elif nparray.dtype == np.int32:
        dn = Array.CreateInstance(Int32, dims_dn)
    elif nparray.dtype == np.bool:
        dn = Array.CreateInstance(Boolean, dims_dn)
    else:
        raise NotImplementedError("dotnetArrayOf does not yet support dtype=" + str(nparray.dtype))
    copyNumpyToDotNetArray(nparray, dn)
    return dn

######################## Conversions from .NET ###########################
    
def numpyArrayOf(dotnetarray):
    ''' Creates and returns a numpy Array that mirrors the provided .NET array
    
    @dotnetarray: .NET Array
    
    @Returns: numpy Array with dtype matching dotnetarray's element type, and identical dimensions with content that matches the provided .NET Array
    '''
    dims = np.empty(dotnetarray.Rank, dtype=int)
    for i in range(dotnetarray.Rank):
        dims[i] = dotnetarray.GetLength(i)
    type_dn = dotnetarray.GetType().GetElementType().Name
    if type_dn == 'Double':
        dtype = float
    elif type_dn == 'Int32':
        dtype = np.int32
    elif type_dn == 'Int64':
        dtype = int #NOTE: probably invalid in Python 3.x
    elif type_dn == 'Boolean':
        dtype = np.bool
    else:
        raise NotImplementedError("numpyArrayOf does not yet support .NET Arrays with " + str(type_dn) + " elements")
    nparray = np.empty(dims, dtype=dtype, order='C')
    copyDotNetArrayToNumpy(dotnetarray, nparray)
    return nparray

.NET:

    public class PythonInterop
    {
        /// <summary>
        /// Copies data from a NumPy array to the destination .NET array
        /// </summary>
        /// <param name="pSource">Pointer to a NumPy array to copy from</param>
        /// <param name="dest">.NET array to be copied into</param>
        /// <remarks>This routine handles Boolean arrays in a special way because NumPy arrays have each element occupying 1 byte while .NET has them occupying 4 bytes</remarks>
		public static void CopyFromPointer(IntPtr pSource, Array dest)
        {
            Type elementType = dest.GetType().GetElementType();
            int sizeOfElement = Marshal.SizeOf(elementType);
            if (elementType == typeof(Boolean))
                sizeOfElement = 1;

            int byteCount = sizeOfElement;
            for (int i = 0; i < dest.Rank; i++)
            {
                byteCount *= dest.GetLength(i);
            }

            var gch = GCHandle.Alloc(dest);
            var tPtr = Marshal.UnsafeAddrOfPinnedArrayElement(dest, 0);
            MemCopy(pSource, tPtr, byteCount);
            gch.Free();
        }

        /// <summary>
        /// Copies data from a .NET array to a NumPy array
        /// </summary>
        /// <param name="source">.NET array to be copied from</param>
        /// <param name="pDest">Pointer to a NumPy array to copy into</param>
		public static void CopyToPointer(Array source, IntPtr pDest)
        {
            Type elementType = source.GetType().GetElementType();
            int sizeOfElement = Marshal.SizeOf(elementType);
            if (elementType == typeof(Boolean))
                sizeOfElement = 1;

            int byteCount = sizeOfElement;
            for (int i = 0; i < source.Rank; i++)
                byteCount *= source.GetLength(i);

            var gch = GCHandle.Alloc(source);
            var tPtr = Marshal.UnsafeAddrOfPinnedArrayElement(source, 0);
            MemCopy(tPtr, pDest, byteCount);
            gch.Free();
        }

        private static readonly int SIZEOFINT = Marshal.SizeOf(typeof(int));
        private static unsafe void MemCopy(IntPtr pSource, IntPtr pDest, int byteCount)
        {
            int count = byteCount / SIZEOFINT;
            int rest = byteCount % count;
            unchecked
            {
                int* ps = (int*)pSource.ToPointer(), pd = (int*)pDest.ToPointer();
                // Loop over the cnt in blocks of 4 bytes, 
                // copying an integer (4 bytes) at a time:
                for (int n = 0; n < count; n++)
                {
                    *pd = *ps;
                    pd++;
                    ps++;
                }
                // Complete the copy by moving any bytes that weren't moved in blocks of 4:
                if (rest > 0)
                {
                    byte* ps1 = (byte*)ps;
                    byte* pd1 = (byte*)pd;
                    for (int n = 0; n < rest; n++)
                    {
                        *pd1 = *ps1;
                        pd1++;
                        ps1++;
                    }
                }
            }
        }
    }

@den-run-ai
Copy link
Contributor

@BenjaminPelletier if you would like to submit a helper function for numpy arrays, then pull requests are welcome! Here is another example of passing numpy arrays:

https://stackoverflow.com/questions/43910749/efficiently-convert-system-single-to-numpy-array

@yagweb
Copy link
Contributor

yagweb commented Jul 25, 2017

@denfromufa I also have a numpy array converter here
https://github.com/yagweb/pythonnetLab/blob/master/pynetLab/Numpy.cs

With this converter, the example https://github.com/pythonnet/pythonnet#example can be replaced with this one,
https://github.com/yagweb/pythonnetLab/blob/master/Test/TestMatplotlib.cs
This one may be a better an example, because,

  1. .NET users can create a numpy array with a single line, like this,
    var x = Numpy.NewArray(new double[]{ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0 });
    So, No CLR list converting to python list by default is needed.

  2. Usage of PyScope and Matplotlib is also included in this example.

...
scope.Exec(
    "fig = plt.figure() \n" +
    "plt.plot(x[1:], y[1:]) \n"   //we can slice a numpy array
);
var fig = scope.Get("fig");
//fig.show(); //show the figure
plotdata = Matplotlib.SaveFigureToArray(fig, 200, "png"); //save the figure to a .NET array

@den-run-ai
Copy link
Contributor

@yagweb do you suggest that we should revert back the auto-conversion due to this regression and use your numpy converter?

@vmuriart
Copy link
Contributor

which commit broke this ? Was it the one I added to fix the example?
If my commit broke existing behavior, then I'm ok w reverting that one and reopenning the other issues.

@den-run-ai
Copy link
Contributor

den-run-ai commented Aug 10, 2017

@vmuriart this pull request broke C# List behavior:

https://github.com/pythonnet/pythonnet/pull/427/files

@vmuriart
Copy link
Contributor

ok to revert behavior until better solution is found

@robbmcleod
Copy link

robbmcleod commented Dec 8, 2017

Hi, I wrote a Python-side Numpy <-> .NET conversion functions that use ctypes.memmove based on what @BenjaminPelletier provided above and the 2014 mailing list conversation between David Cook and Jeffrey Bush. It doesn't require any C# helper code. I think I've got support for all the types in System except strings. Performance seems adequate, I can do around 1 MB / ms, after the initial lazy import:

Numpy to .NET converted 8388608 bytes in 4.231 +/- 1.812 ms (mean: 4.0 ns/ele)
.NET to Numpy converted 8388608 bytes in 3.461 +/- 1.200 ms (mean: 3.3 ns/ele)

The ctypes.memmove approach is equally as fast as Marshal.Copy and doesn't have any issues with multi-dimensional arrays associated with it. Source code as follows, the __main__ block is all testing code:

import numpy as np
import ctypes
import clr, System
from System import Array, Int32
from System.Runtime.InteropServices import GCHandle, GCHandleType

_MAP_NP_NET = {
    np.dtype('float32'): System.Single,
    np.dtype('float64'): System.Double,
    np.dtype('int8')   : System.SByte,
    np.dtype('int16')  : System.Int16,
    np.dtype('int32')  : System.Int32,
    np.dtype('int64')  : System.Int64,
    np.dtype('uint8')  : System.Byte,
    np.dtype('uint16') : System.UInt16,
    np.dtype('uint32') : System.UInt32,
    np.dtype('uint64') : System.UInt64,
    np.dtype('bool')   : System.Boolean,
}
_MAP_NET_NP = {
    'Single' : np.dtype('float32'),
    'Double' : np.dtype('float64'),
    'SByte'  : np.dtype('int8'),
    'Int16'  : np.dtype('int16'), 
    'Int32'  : np.dtype('int32'),
    'Int64'  : np.dtype('int64'),
    'Byte'   : np.dtype('uint8'),
    'UInt16' : np.dtype('uint16'),
    'UInt32' : np.dtype('uint32'),
    'UInt64' : np.dtype('uint64'),
    'Boolean': np.dtype('bool'),
}

def asNumpyArray(netArray):
    '''
    Given a CLR `System.Array` returns a `numpy.ndarray`.  See _MAP_NET_NP for 
    the mapping of CLR types to Numpy dtypes.
    '''
    dims = np.empty(netArray.Rank, dtype=int)
    for I in range(netArray.Rank):
        dims[I] = netArray.GetLength(I)
    netType = netArray.GetType().GetElementType().Name

    try:
        npArray = np.empty(dims, order='C', dtype=_MAP_NET_NP[netType])
    except KeyError:
        raise NotImplementedError("asNumpyArray does not yet support System type {}".format(netType) )

    try: # Memmove 
        sourceHandle = GCHandle.Alloc(netArray, GCHandleType.Pinned)
        sourcePtr = sourceHandle.AddrOfPinnedObject().ToInt64()
        destPtr = npArray.__array_interface__['data'][0]
        ctypes.memmove(destPtr, sourcePtr, npArray.nbytes)
    finally:
        if sourceHandle.IsAllocated: sourceHandle.Free()
    return npArray

def asNetArray(npArray):
    '''
    Given a `numpy.ndarray` returns a CLR `System.Array`.  See _MAP_NP_NET for 
    the mapping of Numpy dtypes to CLR types.

    Note: `complex64` and `complex128` arrays are converted to `float32` 
    and `float64` arrays respectively with shape [m,n,...] -> [m,n,...,2]
    '''
    dims = npArray.shape
    dtype = npArray.dtype
    # For complex arrays, we must make a view of the array as its corresponding 
    # float type.
    if dtype == np.complex64:
        dtype = np.dtype('float32')
        dims.append(2)
        npArray = npArray.view(np.float32).reshape(dims)
    elif dtype == np.complex128:
        dtype = np.dtype('float64')
        dims.append(2)
        npArray = npArray.view(np.float64).reshape(dims)

    netDims = Array.CreateInstance(Int32, npArray.ndim)
    for I in range(npArray.ndim):
        netDims[I] = Int32(dims[I])
    
    if not npArray.flags.c_contiguous:
        npArray = npArray.copy(order='C')
    assert npArray.flags.c_contiguous

    try:
        netArray = Array.CreateInstance(_MAP_NP_NET[dtype], netDims)
    except KeyError:
        raise NotImplementedError("asNetArray does not yet support dtype {}".format(dtype))

    try: # Memmove 
        destHandle = GCHandle.Alloc(netArray, GCHandleType.Pinned)
        sourcePtr = npArray.__array_interface__['data'][0]
        destPtr = destHandle.AddrOfPinnedObject().ToInt64()
        ctypes.memmove(destPtr, sourcePtr, npArray.nbytes)
    finally:
        if destHandle.IsAllocated: destHandle.Free()
    return netArray

if __name__ == '__main__':
    from time import perf_counter
    import matplotlib.pyplot as plt
    import psutil

    tries = 1000
    foo = np.full([1024,1024], 2.5, dtype='float32')


    netMem = np.zeros(tries)
    t_asNet = np.zeros(tries)
    netFoo = asNetArray( foo ) # Lazy loading makes the first iteration very slow
    for I in range(tries):
        t0 = perf_counter()
        netFoo = asNetArray( foo )
        t_asNet[I] = perf_counter() - t0
        netMem[I] = psutil.virtual_memory().free / 2.0**20

    t_asNumpy = np.zeros(tries)
    numpyMem = np.zeros(tries)
    unNetFoo = asNumpyArray( netFoo ) # Lazy loading makes the first iteration very slow
    for I in range(tries):
        t0 = perf_counter()
        unNetFoo = asNumpyArray( netFoo )
        t_asNumpy[I] = perf_counter() - t0
        numpyMem[I] = psutil.virtual_memory().free / 2.0**20

    # Convert times to milliseconds
    t_asNet *= 1000
    t_asNumpy *= 1000
    np.testing.assert_array_almost_equal( unNetFoo, foo )
    print( "Numpy to .NET converted {} bytes in {:.3f} +/- {:.3f} ms (mean: {:.1f} ns/ele)".format( \
        foo.nbytes, t_asNet.mean(), t_asNet.std(), t_asNet.mean()/foo.size*1e6 ) )
    print( ".NET to Numpy converted {} bytes in {:.3f} +/- {:.3f} ms (mean: {:.1f} ns/ele)".format( \
        foo.nbytes, t_asNumpy.mean(), t_asNumpy.std(), t_asNumpy.mean()/foo.size*1e6 ) )

    plt.figure()
    plt.plot(np.arange(tries), netMem, '-', label='asNetArray')
    plt.plot(np.arange(tries), numpyMem, '-', label='asNumpyArray')
    plt.legend(loc='best')
    plt.ylabel('Free memory (MB)')
    plt.xlabel('Iteration')
    plt.show(block=True)

I don't see any evidence of memory-leaking.

Edit: one can do a zero-copy with np.frombuffer but then you have a mess of memory manged both by Python's garbage collector and C#'s garbage collector. If people here know how to deal with references in both GC's let me know.

@den-run-ai
Copy link
Contributor

@robbmcleod I just wanted to say thanks for your code and give feedback - it works great! I especially appreciate not having .NET wrapper and handling all conversion in Python/pythonnet/ctypes.

@den-run-ai den-run-ai changed the title Implicit List conversion breaks backwards compatibility Implicit List conversion breaks backwards compatibility + numpy support Jan 31, 2018
@den-run-ai den-run-ai added this to the 2.4.0 milestone Nov 5, 2018
@filmor filmor modified the milestones: 2.4.0, 2.5.0 Apr 12, 2019
@fartsmajeure
Copy link

I have a use-case that is also broken by the #427 change and doesn't have a good workaround.
Consider the following C# class:

namespace MyNamespace
{
    public class MyListContainerClass
    {
        private List<double> _myList = new List<double>();

        public List<double> MyList
        {
            get { return _myList; }
        }
    }
}

Once the List is converted to a python list, any mutations to the python list don't get updated in the original C# class.
You can see this with the following python code:

my_test_class = MyNamespace.MyListContainerClass()
my_list = my_test_class.MyList

# throws exception
# my_list.Add(4)

my_list.append(4)
assert (len(my_list) == 1) # passes

my_list = my_test_class.MyList
assert (len(my_list) == 1) # fails!!!!!

This breaks any case where you would have some other C# method on the class that needs to use the list after it's been modified by python.

@lostmsu
Copy link
Member

lostmsu commented Sep 12, 2019

Tangentially related, but in our fork I just added a way to extend how .NET types are converted to Python and vice versa (so far without disabling conversions, that are coming out of the box, but I am planning this too).

See losttech@c0ffc29

lostmsu added a commit to losttech/pythonnet that referenced this issue Apr 23, 2020
Now Python host can force raw encoding for autoconverted .NET types.
Enables workaround for pythonnet#514
lostmsu added a commit to losttech/pythonnet that referenced this issue Apr 23, 2020
Now Python host can force raw encoding for autoconverted .NET types.
Enables workaround for pythonnet#514
@lostmsu lostmsu mentioned this issue Apr 23, 2020
4 tasks
lostmsu added a commit to losttech/pythonnet that referenced this issue Apr 23, 2020
Now Python host can force raw encoding for autoconverted .NET types.
Enables workaround for pythonnet#514
@lostmsu
Copy link
Member

lostmsu commented Apr 27, 2020

The new RawProxyEncoder from #1122 could be used to work around this issue.

filmor pushed a commit that referenced this issue Apr 29, 2020
Now Python host can force raw encoding for autoconverted .NET types.
Enables workaround for #514
@lostmsu
Copy link
Member

lostmsu commented Apr 30, 2020

This should now be solved by codecs: Codecs: .NET <-> Python conversions

@lostmsu lostmsu closed this as completed Apr 30, 2020
@robbmcleod
Copy link

Thanks.

AlexCatarino pushed a commit to QuantConnect/pythonnet that referenced this issue Jun 29, 2020
Now Python host can force raw encoding for autoconverted .NET types.
Enables workaround for pythonnet#514
@ddsleonardo
Copy link

@lostmsu Hi there. Would you be able to provide an example of how to use Codecs to prevent the conversion from a System.Collections.Generic.List to a python list? I read both the Codec wiki page and this example but I didn't get it, sorry.
Is there a way to just set this behaviour by default anytime I import pythonnet?

@jcarole
Copy link

jcarole commented Oct 30, 2020 via email

@ddsleonardo
Copy link

I do not understand your reply @jcarole.
To make it clear, I am having the same problem as #1153 and also mentioned here. In both instances there are mentions either of RawProxyEncoder or the Codecs but I could not understand the usage examples. The only working solution I have found is this comment previously made in this discussion. This comment sounded encouraging, it would be really nice if we could make the conversions optional.

@lostmsu
Copy link
Member

lostmsu commented Oct 30, 2020

@ddsleonardo have you checked the new test named test_codec (it is at the very end of the change linked above)? What part you did not understand?

@jcarole
Copy link

jcarole commented Oct 30, 2020 via email

@ddsleonardo
Copy link

@lostmsu Thanks for the reply. My main question was relative to at what scope should I use the Codecs, i.e. do I need to define and register an instance of the Encoder class every time I open a Python Interactive and want to manipulate some C# Lists? (which I believe is the case, correct)?

I think I managed to follow test_codec this time and at least could run the steps without fail. But then I realised that I am running my own build of pythonnet now (with the commented block), so my auto-conversion is disabled anyway.
I will revert this later and give it another go.

Still following on the same question, it sounds like this has been made a lot more convoluted than maybe it should be. I had been running a previous version of pythonnet for a while to avoid this, and only encountered the issue again as I had to reset my environment. Now I appreciate that this package isn't written just for me, but I'd be interested to know the motivations behind this, as it seems that I'm not the only one who has struggled with this and the workarounds to make it work are not trivial.

At the same time, I am happy to run my own fork of this repo without the conversion, and set up a pipeline that keeps everything else up to date with the main repo.

@lostmsu
Copy link
Member

lostmsu commented Oct 30, 2020

@ddsleonardo don't worry, the default conversion is going to be off by default in the next major version.

I was not here around the time Python.NET started, but I suspect this conversion was added to make most APIs easily callable: List<T> does not implement Python collection protocol, and Python's list does not implement IEnumerable and other stuff.

@robbmcleod
Copy link

Updated my code snippet for PythonNet 2.5.1, Python 3.9.4, and NumPy 1.20.2 for those who are looking for a working solution:

https://gist.github.com/robbmcleod/73ca42da5984e6d0e5b6ad28bc4a504e

@CitizenInsane
Copy link

@robbmcleod I adapted your last code so that complex arrays in python convert back-and-forth with SystemArray in .NET whose element type are directly System.Numerics.Complex (instead of adding extra dimension):

https://gist.github.com/CitizenInsane/c8d3ddc5b14faceec433af3e940e22a8

@robbmcleod
Copy link

@CitizenInsane thanks although in my case I actually do need the 64-bit complex as that's what the counterparty software that I'm talking to uses.

@CitizenInsane
Copy link

@robbmcleod Yes it all depends. The .NET API i try to connect with is using System.Numerics.Complex arrays directly. Thanks for the great job, first time I'm doing python and this to interop with .net so it helped a lot to have your converters :).

@lostmsu
Copy link
Member

lostmsu commented Sep 23, 2021

BTW, @robbmcleod , @CitizenInsane , in pythonnet 3.0 .NET arrays will implement buffer protocol!

@robbmcleod
Copy link

BTW, @robbmcleod , @CitizenInsane , in pythonnet 3.0 .NET arrays will implement buffer protocol!

Good to know, thanks, that should be a big improvement over my memcpy solution.

@Jiew23
Copy link

Jiew23 commented Dec 23, 2022

@denfromufa I also have a numpy array converter here https://github.com/yagweb/pythonnetLab/blob/master/pynetLab/Numpy.cs

With this converter, the example https://github.com/pythonnet/pythonnet#example can be replaced with this one, https://github.com/yagweb/pythonnetLab/blob/master/Test/TestMatplotlib.cs This one may be a better an example, because,

  1. .NET users can create a numpy array with a single line, like this,
    var x = Numpy.NewArray(new double[]{ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0 });
    So, No CLR list converting to python list by default is needed.
  2. Usage of PyScope and Matplotlib is also included in this example.
...
scope.Exec(
    "fig = plt.figure() \n" +
    "plt.plot(x[1:], y[1:]) \n"   //we can slice a numpy array
);
var fig = scope.Get("fig");
//fig.show(); //show the figure
plotdata = Matplotlib.SaveFigureToArray(fig, 200, "png"); //save the figure to a .NET array

@yagweb Hi there, I found that in your converter, "using(Py.GIL())" is called multiple times in a function. However, every time that I call the function on my code for the second time, the code just hangs at "using(Py.GIL())". I have to restart the whole program to get the function to work for the second time. Could you please advise how the system should be set up to solve this problem? Thank you so much!

@Kemsekov
Copy link

Kemsekov commented Jan 4, 2024

Thanks
@robbmcleod

I decided to add missing part of reply in regards to how convert numpy array from c# side.

All that left to do is load your script as module and call these functions.

Copy script that robbmcleod posted and save as array_converter.py in your directory

Put this into c# project

using Python.Runtime;
public static class NumpyUtils
{
    public static string script = File.ReadAllText("array_converter.py");
    public static dynamic ArrayConverterModule(){
        return PyModule.FromString("",script);
    }
}

Call it from c# and cast types right away

dynamic np_array = ...; //some numpy float32 array
dynamic array_converter = NumpyUtils.ArrayConverterModule();
float[] converted  array_converter.asNetArray(np_array);

Also I would like to add script that I use for slicing numpy arrays

using Python.Runtime;

public static class NumpyUtils
{
    public static string script = File.ReadAllText("array_converter.py");
    public static dynamic ArrayConverterModule(){
        return PyModule.FromString("",script);
    }
    /// <summary>
    /// Slices numpy array
    /// </summary>
    /// <param name="slice">Slice without object name and brackets []. For example : ":5,3"</param>
    public static dynamic Slice(PyObject np_array,string slice)
    {
        using var glob = new PyDict();
        glob["arr"] = np_array;
        var sliced = PythonEngine.Eval($"arr[{slice}]",
            globals: glob
        );
        return sliced;
    }
}
dynamic np_array = ...; //some numpy float32 array
var firstFive = NumpyUtils.Slice(np_array,":5");

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests