Skip to content

Use Python buffer protocol when converting Python objects to .NET arrays #1838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lostmsu opened this issue Jun 27, 2022 · 9 comments
Open

Comments

@lostmsu
Copy link
Member

lostmsu commented Jun 27, 2022

Copying data from NumPy array to C# is a common task. Python.NET implements a simple sequence-to-array copying mechanism, which uses iterators. It is extremely inefficient for large arrays of primitive types.

Now we have a better way - using PyBuffer and buffer protocol. We could detect, that Python object implements the buffer protocol, use it to copy data when possible, and fallback to iteration otherwise.

@lostmsu
Copy link
Member Author

lostmsu commented Jun 27, 2022

If somebody wants to take on it, this test shows simple use of PyBuffer: TestBufferRead

And this is the method that needs to be changed: Converter.ToArray

@filmor
Copy link
Member

filmor commented Jun 27, 2022

This would not break API/ABI, so we can integrate it in a normal point release after 3.0, right?

@lostmsu
Copy link
Member Author

lostmsu commented Jun 27, 2022

Yep.

@lostmsu lostmsu added this to the PRs welcome milestone Jun 28, 2022
@bruceYangxianglin
Copy link

Copying data from C# array to Python list or Numpy array is also a common task. Is there a more efficient way?

@bjudkewitz
Copy link

Agreed, converting a python array to a C# array is a very common task, and it is currently extremely slow. It would be fantastic to make it faster. Is it currently possibly with Python (and Python.NET) only, without any C# programming?

@yahav6893
Copy link

Hi! I’m looking for some guidance on how to speed up copying from Python to C#.
Would anyone be able to point me in the right way to do that?
Thanks ahead.

@chickenservice
Copy link

Hi, is someone currently working on this? If not I'd be happy to try. I've been using pythonnet at work successfully for years and would be glad to help. I'm familiar with the topic because I have already dealt with converting numpy arrays from/to C# arrays in an efficient manner. It'd be great to have this natively in pythonnet.

@filmor
Copy link
Member

filmor commented Apr 15, 2024

@chickenservice To my knowledge, no one is currently actively working on this, so give it a shot! An implementation would be greatly appreciated!

@chickenservice
Copy link

@filmor Perfect, is there anything besides the steps in CONTRIBUTING.md that I need to be aware of?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants