1838 Use Python buffer protocol when converting Python objects to .NET arrays #2372

chickenservice · 2024-05-05T14:46:16Z

What does this implement/fix? Explain your changes.

If a python object exposes the buffer protocol then copy the buffer directly to a .NET array. This works for (blittable) single- or multidimensional arrays. I added tests for numpy arrays, bytearrays and python arrays to check whether this change works across different buffer protocol implementations.

My implementation currently has the following caveats:

The Python object must implement the buffer protocol and be C-contiguous. This is only relevant for multidimensional arrays. I explicitly request a C-contiguous buffer because (multidimensional) .NET arrays are C-contiguous as well. This allows copying the whole chunk of memory at once. Supporting multidimensional Fortran-contiguous buffers would make the implementation more complicated though I have not thought too much about it.
PIL-style buffers that do not store the whole buffer contiguously but through pointers via suboffsets are not supported. This would also need special handling.
The .NET element type of the array must be blittable and not a struct. Structs can be blittable if all their members are blittable. I imagine this may make the implementation more complicated because as far as I know no there's no out-of-the-box way to check whether a type is blittable. I think this is better handled with custom Codecs?
Works for multidimensional arrays which ToArray previously didn't handle, so we weaken a precondition of the converter. However efficient copying of multidimensional arrays is highly desirable in my opinion.
I use Buffer.MemoryCopy which is marked as not CLS-compliant. If this is not acceptable I will of course find another way.
The implementation is done directly with the C-API. I can rewrite this with PyBuffer if you want.

Does this close any currently open issues?

1838 Use Python buffer protocol when converting Python objects to .NET arrays

Any other comments?

The test with bytearray to .NET byte array is currently failing for NET 6.0 due to the way the Py_Buffer struct is being marshalled. I opened a separate issue #2371 to address this.

Checklist

Check all those that are applicable and complete.

Make sure to include one or more tests for your change
If an enhancement PR, please create docs and at best an example
Ensure you have signed the .NET Foundation CLA
Add yourself to AUTHORS
Updated the CHANGELOG

If a python object is converted to a managed array and it exposes the buffer protocol the buffer can be copied directly to the managed array. For now this works only if the requested array element type is blittable.

Currently not handled by the buffer copying mechanism anyway

lostmsu · 2024-05-06T06:56:59Z

src/runtime/Converter.cs

        /// </summary>
        private static bool ToArray(BorrowedReference value, Type obType, out object? result, bool setError)
        {
            Type elementType = obType.GetElementType();
            result = null;

+            // structs can also be blittable but for the standard usecases this suffices
+            var isBlittable = elementType.IsPrimitive && elementType != typeof(char) && elementType != typeof(bool);
+           if (isBlittable && Runtime.PyObject_GetBuffer(value, out var view, (int)PyBUF.CONTIG_RO) == 0)


Both directions need a check that source and target element types match.

Would it be ok to match the typecode of the .NET type to the corresponding format string of the buffer, i.e. roughly like so:

var compatible = false; var format = buf.format; switch (tc) { case TypeCode.Boolean: compatible = format == "?"; break; case TypeCode.Byte: compatible = format == "b"; break; //etc... default: break;

This would require #2371 to be fixed so I can actually retrieve the format field.

lostmsu · 2024-05-06T07:01:00Z

src/runtime/Converter.cs

        /// </summary>
        private static bool ToArray(BorrowedReference value, Type obType, out object? result, bool setError)
        {
            Type elementType = obType.GetElementType();
            result = null;

+            // structs can also be blittable but for the standard usecases this suffices
+            var isBlittable = elementType.IsPrimitive && elementType != typeof(char) && elementType != typeof(bool);
+           if (isBlittable && Runtime.PyObject_GetBuffer(value, out var view, (int)PyBUF.CONTIG_RO) == 0)


I'm not sure it is safe to assume getting buffer has the same semantics as using iterator for an arbitrary type that supports GetBuffer.

Imho, this could be made the default behavior for converting a fixed set of standard types, but for the others it might be better to expose some kind of explicit call instead of conversion. E.g. pyObjectInst.ReadBuffer(destinationArray)

Isn't this caught by the primitive blitable check?

We could move the whole thing to after the if (IterObject.IsNull()) check to make sure it's an iterable? Other than that as @filmor said I already allow only primitive blittable types as array element types.

This issue is not about element types, but about container types. Does Python guarantee, that GetBuffer()[0], GetBuffer()[1], etc is the same sequence as .iter[0], .iter[1], etc for every type that implements GetBuffer?

I thought that was the point of the buffer protocol :). I'll read up more on it.

@chickenservice the concern is about breaking changes, especially within Python.NET 3.x, more especially within Python.NET 3.0.x.

If there's a difference in result, we should delay the change of behavior to 4.x, and in 3.x only add new functions.

IMHO, we can only immediately apply the speedups for the types that we know have the same behavior.

@lostmsu that makes sense. The tests cover bytearray, array.array and ndarray. From my experience these behave according to what you described and are the canonical implementations anyway. Any concerns? Are there other types you have in mind? Should we limit ndim == 1 for now?

If the tests do not cover multidim or the behavior in multidim is different, we should limit ndim to 1.

But again, I strongly suggest to add an explicit new method to PyObject that does the right thing with buffers, and only works with buffer protocol.

I think I misunderstood your suggestion, to clarify:

For upcoming minor/patch release expose this as a dedicated method and remove it from ToArray completely as breaking changes cannot be ruled out completely.

For the next major release reevaluate whether this can/should be done implicitly in ToArray and accept breaking changes.

@chickenservice yep, that's exactly right.

vadimkantorov · 2024-08-06T11:40:08Z

My related attempts / investigation on ways of NumPy <> C# arrays interop:

How to pass a numpy float array from Python to C#? #2226
I also discovered a zero-copy way of NumPy -> C# array interop: https://gist.github.com/vadimkantorov/c9ab97aff995ea01ea7f6248ddc733bc#file-csharpfrompython-cs-L23-L41 using some sort of System.Buffers.MemoryManager device
zero-copy is also useful as it can enable inplace modification of NumPy arrays in C# - which can be useful for really large arrays and keeping all input/output allocs in Python/NumPy land

cc @myrthel

lostmsu · 2025-04-11T04:51:56Z

Closing as the requested changes were not implemented.

chickenservice added 3 commits April 24, 2024 22:42

Use buffer protocol to convert python objects to managed arrays

1d86ad4

If a python object is converted to a managed array and it exposes the buffer protocol the buffer can be copied directly to the managed array. For now this works only if the requested array element type is blittable.

Add tests for python arrays

b487188

Remove bytearray to char test

2e2866d

Currently not handled by the buffer copying mechanism anyway

lostmsu requested changes May 6, 2024

View reviewed changes

lostmsu closed this Apr 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1838 Use Python buffer protocol when converting Python objects to .NET arrays #2372

1838 Use Python buffer protocol when converting Python objects to .NET arrays #2372

chickenservice commented May 5, 2024

lostmsu May 6, 2024

chickenservice May 6, 2024 •

edited

Loading

lostmsu May 6, 2024

filmor May 6, 2024

chickenservice May 6, 2024 •

edited

Loading

lostmsu May 6, 2024 •

edited

Loading

filmor May 6, 2024

lostmsu May 7, 2024

chickenservice May 9, 2024

lostmsu May 10, 2024 •

edited

Loading

chickenservice May 11, 2024 •

edited

Loading

lostmsu May 11, 2024

vadimkantorov commented Aug 6, 2024 •

edited

Loading

lostmsu commented Apr 11, 2025

1838 Use Python buffer protocol when converting Python objects to .NET arrays #2372

1838 Use Python buffer protocol when converting Python objects to .NET arrays #2372

Conversation

chickenservice commented May 5, 2024

What does this implement/fix? Explain your changes.

Does this close any currently open issues?

Any other comments?

Checklist

lostmsu May 6, 2024

Choose a reason for hiding this comment

chickenservice May 6, 2024 • edited Loading

Choose a reason for hiding this comment

lostmsu May 6, 2024

Choose a reason for hiding this comment

filmor May 6, 2024

Choose a reason for hiding this comment

chickenservice May 6, 2024 • edited Loading

Choose a reason for hiding this comment

lostmsu May 6, 2024 • edited Loading

Choose a reason for hiding this comment

filmor May 6, 2024

Choose a reason for hiding this comment

lostmsu May 7, 2024

Choose a reason for hiding this comment

chickenservice May 9, 2024

Choose a reason for hiding this comment

lostmsu May 10, 2024 • edited Loading

Choose a reason for hiding this comment

chickenservice May 11, 2024 • edited Loading

Choose a reason for hiding this comment

lostmsu May 11, 2024

Choose a reason for hiding this comment

vadimkantorov commented Aug 6, 2024 • edited Loading

lostmsu commented Apr 11, 2025

chickenservice May 6, 2024 •

edited

Loading

chickenservice May 6, 2024 •

edited

Loading

lostmsu May 6, 2024 •

edited

Loading

lostmsu May 10, 2024 •

edited

Loading

chickenservice May 11, 2024 •

edited

Loading

vadimkantorov commented Aug 6, 2024 •

edited

Loading