Codecs: customize set of Py<->C# conversions via PyObjectConversions #1022

lostmsu · 2019-12-19T22:45:35Z

What does this implement/fix? Explain your changes.

This enables registering new automatic Py->C# and C#->Py conversions (codecs), that will run when Python calls .NET code and vice versa.

Does this close any currently open issues?

This is a new feature

Any other comments?

...

Checklist

Check all those that are applicable and complete.

Make sure to include one or more tests for your change
If an enhancement PR, please create docs and at best an example: see TupleCodec
Add yourself to AUTHORS
Updated the CHANGELOG

koubaa · 2019-12-21T16:33:13Z

src/embed_tests/Python.EmbeddingTest.15.csproj

@@ -23,7 +23,7 @@
    <SolutionDir Condition="$(SolutionDir) == '' Or $(SolutionDir) == '*Undefined*'">..\..\</SolutionDir>
    <PythonBuildDir Condition="'$(TargetFramework)'=='net40' AND '$(PythonBuildDir)' == ''">$(SolutionDir)\bin\</PythonBuildDir>
    <PublishDir Condition="'$(TargetFramework)'!='net40'">$(OutputPath)\$(TargetFramework)_publish</PublishDir>
-    <LangVersion>6</LangVersion>
+    <LangVersion>7.3</LangVersion>


Is this intentional? @lostmsu

Yes, I think I am using a few features. Is there a C# compiler, that does not support 7.3 yet?
It is 7.3 in Runtime for a few months now.

@lostmsu just wanted to make sure it wasn't added by mistake.

koubaa · 2020-01-15T23:18:48Z

src/runtime/runtime.cs

+        /// <summary>
+        /// Increase Python's ref counter for the given object, and get the object back.
+        /// </summary>
+        internal static IntPtr SelfIncRef(IntPtr op)


This is neat and might simplify other code in the project

koubaa · 2020-01-15T23:19:24Z

I like the conciseness of the terminology (encode/decode) but for me it isn't entirely obvious which direction (py <-> C#) is encoding and decoding. It is ok with me as long as we document it well enough.

Which other encoders should exist inside of Python.Runtime aside from the Tuple one you've added? Would you be OK with the List and Action ones I am interested in eventually being distributed by PythonNet?

I am writing the below snippet to document how it'd be available for external codecs to see how we like the syntax. To me it is not too bad:

import clr
clr.AddReference("MyCodecs")
import MyCodecs
for encoder in MyCodecs.Encoders:
    Python.Runtime.PyObjectConversions.RegisterEncoder(encoder)
for decoder in MyCodecs.Decoders:
    Python.Runtime.PyObjectConversions.RegisterDecoder(decoder)

koubaa · 2020-01-15T23:22:35Z

src/runtime/converter.cs

@@ -135,8 +135,16 @@ internal static IntPtr ToPython(object value, Type type)
                return result;
            }

-			if (value is IList && !(value is INotifyPropertyChanged) && value.GetType().IsGenericType)
-			{
+            if (Type.GetTypeCode(type) == TypeCode.Object && value.GetType() != typeof(object)) {


glad this comes before other builtin conversions but ultimately the built-in conversions should probably use this codec system internally.

src/runtime/converterextensions.cs

lostmsu · 2020-01-16T19:08:05Z

Pushed a fix to failing test

filmor · 2020-01-17T16:42:29Z

src/runtime/Codecs/TupleCodecs.cs

+
+        public bool CanEncode(Type type)
+            => type.Namespace == typeof(TTuple).Namespace && type.Name.StartsWith(typeof(TTuple).Name + '`')
+            || type == typeof(object) || type == typeof(TTuple);


Maybe write this one out as a normal function block, it's a bit tedious to read.

Also, can't you just use type.IsGenericType && type.GetGenericTypeDefinition() == typeof(TTuple) or something like that?

added sample TupleCodec (only supporting ValueTuple)

…ion taking priority

lostmsu · 2020-01-30T17:31:48Z

Mark methods as obsolete until they are stable.

…in Python 2.7 x64

codecov-io · 2020-02-01T07:27:53Z

Codecov Report

Merging #1022 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1022   +/-   ##
=======================================
  Coverage   86.75%   86.75%           
=======================================
  Files           1        1           
  Lines         302      302           
=======================================
  Hits          262      262           
  Misses         40       40

Flag	Coverage Δ
#setup_linux	`65.56% <ø> (ø)`	⬆️
#setup_windows	`71.52% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5619fb9...41de69d. Read the comment docs.

lostmsu · 2020-02-13T02:28:28Z

@filmor I think this can be merged

… as the decoded object might store a reference to the original PyObject

…return a new reference

lostmsu · 2020-02-26T22:36:32Z

Either I got lucky in this run, or #1050 actually was the real cause for test failures in this branch, and so was fixed by #1062.

src/embed_tests/TestPyScope.cs

amos402 · 2020-02-28T07:47:14Z

The code style of this PR is totally different by the current code, is it really good for this?

igiona · 2021-09-16T20:18:54Z

The issue #823 has been marked as solved by this PR.
It's not clear to me which approach has been taken here (from the ones described in the mentioned bug)?
@lostmsu is the PR going to make Python.NET fully support .NET objects providing the IDynamicMetaObjectProvider interface? I'm moving away from IronPython (which supports IDynamicMetaObjectProvider objects) and running into many issues...

From what I can see, this new feature didn't make it to the latest release, is there a plan for for a 2.5.3 or there will be only the v3.0.0?
Thanks

lostmsu · 2021-09-16T20:53:36Z

@igiona There's no support for IDynamicMetaObjectProvider on custom .NET objects. Instead for custom types to be presented to Python (or vice versa) as something they are not, you need to implement encoders and/or decoders: customizing object marshalling between .NET and Python

igiona · 2021-09-17T05:33:01Z

Thank you @lostmsu for the quick answer.
This implies that the PythonNet assemblies needs to imported in the .NET application, which is not always possible/desirable (not technically, but rather "politically"/by-design.

What's the rational behind the decision to not support IDynamicMetaObjectProvider ?
Is there some architectural constraint that prevent this, or it has a chance to get supported by Python.NET in the future?

lostmsu · 2021-09-17T16:56:01Z

The rational is that pythonnet is an open source project, and nobody stepped up to implement this feature.

I am not sure what you mean by "PythonNet assemblies needs to imported in the .NET application, which is not always possible/desirable". You can develop and register codecs from Python (though development in Python might be tricky). You can also develop codecs in C#, compile them to a DLL, and register them from it. e.g.

clr.AddReference('MyCodecs.dll')

from MyCodecs import MyMegaCodecForTypeX

codec = MyMegaCodecForTypeX()
PyObjectConversions.RegisterEncoder(codec)
PyObjectConversions.RegisterDecoder(codec)

... work with TypeX ...

igiona · 2021-09-19T14:36:36Z

The rational is that pythonnet is an open source project, and nobody stepped up to implement this feature.

This makes sense 👍

I did not checked yet the implementation above into details, but I assumed that the codecs have to implement the interfaces IPyObjectDecoder and/or IPyObjectEncoder in order to be able to be registered via RegisterEncoder/RegisterDecoder.
Which means that my C#-application (which provides these codec) needs to import the PaythonNET assemblies as dependency in order to compile ("which is not always possible/desirable"). Unless your implementation looks for the interface method names by reflection and wraps/maps them to the PythonNET interfaces internally!?

lostmsu · 2021-09-19T16:59:29Z

Again, the app does not have to import pythonnet assemblies. Codecs can be put into a separate library.

lostmsu force-pushed the PR/Codecs branch 2 times, most recently from e840a4b to 3dff8b5 Compare December 20, 2019 02:20

lostmsu mentioned this pull request Dec 20, 2019

Improve method binding #974

Closed

4 tasks

koubaa reviewed Dec 21, 2019

View reviewed changes

koubaa reviewed Jan 15, 2020

View reviewed changes

koubaa approved these changes Jan 15, 2020

View reviewed changes

koubaa reviewed Jan 15, 2020

View reviewed changes

src/runtime/converterextensions.cs Show resolved Hide resolved

lostmsu requested a review from filmor January 16, 2020 19:05

filmor approved these changes Jan 22, 2020

View reviewed changes

lostmsu force-pushed the PR/Codecs branch from df98205 to a1f4aff Compare January 30, 2020 01:56

lostmsu added 3 commits January 29, 2020 20:12

enable expanding set of marshaling conversions via PyObjectConversions

28143d5

added sample TupleCodec (only supporting ValueTuple)

fixed ConversionsObject test failing due to sequence to array convers…

6eca169

…ion taking priority

attempt to fix CI build issue with ValueTuple under Mono

97e33c7

lostmsu force-pushed the PR/Codecs branch from a1f4aff to 97e33c7 Compare January 30, 2020 04:12

added RefereneAssemblies package reference to fix CI build

449338f

lostmsu added 2 commits January 31, 2020 20:35

marked the new codecs API as unstable

39b2347

attempt to fix PyScopeTest.TestThread() reading stale value from res …

daa2901

…in Python 2.7 x64

lostmsu force-pushed the PR/Codecs branch from 4b94a46 to daa2901 Compare February 1, 2020 06:43

lostmsu and others added 6 commits February 12, 2020 18:28

Merge branch 'master' into PR/Codecs

e8e3b4b

Merge branch 'master' into PR/Codecs

8818610

Merge branch 'master' into PR/Codecs

82f6b99

fixed bad IncRef after PyTuple_New

ec98209

corrected reference counting in Codecs

50a3822

don't dispose encoded object in case codec keeps a cache of them

2e19f2c

filmor and others added 4 commits February 23, 2020 18:05

Merge branch 'master' into PR/Codecs

44bfec2

do not dispose object, that might have been just decoded succesfully,…

399ae54

… as the decoded object might store a reference to the original PyObject

remove incref for tuple fields, as Converter.ToPython is supposed to …

e2d3333

…return a new reference

Merge branch 'master' into PR/Codecs

5619fb9

lostmsu force-pushed the PR/Codecs branch from 04b539e to 5619fb9 Compare February 25, 2020 22:28

Merge branch 'master' into PR/Codecs

41de69d

lostmsu merged commit 4d1442d into pythonnet:master Feb 26, 2020

amos402 reviewed Feb 27, 2020

View reviewed changes

src/embed_tests/TestPyScope.cs Show resolved Hide resolved

This was referenced Feb 27, 2020

Enable user classes to override how Python.NET processes parameters of their functions #835

Closed

TestThread does not synchronize properly #1067

Closed

lostmsu deleted the PR/Codecs branch March 7, 2020 21:19

lostmsu mentioned this pull request May 5, 2020

A way to override Python.NET .NET <-> Python wrapping for custom classes #823

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Codecs: customize set of Py<->C# conversions via PyObjectConversions #1022

Codecs: customize set of Py<->C# conversions via PyObjectConversions #1022

lostmsu commented Dec 19, 2019 •

edited

Loading

koubaa Dec 21, 2019

lostmsu Dec 22, 2019 •

edited

Loading

koubaa Jan 15, 2020

koubaa Jan 15, 2020

koubaa commented Jan 15, 2020

koubaa Jan 15, 2020

lostmsu commented Jan 16, 2020

filmor Jan 17, 2020

filmor Jan 17, 2020

lostmsu commented Jan 30, 2020

codecov-io commented Feb 1, 2020 •

edited

Loading

lostmsu commented Feb 13, 2020

lostmsu commented Feb 26, 2020

amos402 commented Feb 28, 2020

igiona commented Sep 16, 2021

lostmsu commented Sep 16, 2021 •

edited

Loading

igiona commented Sep 17, 2021

lostmsu commented Sep 17, 2021 •

edited

Loading

igiona commented Sep 19, 2021 •

edited

Loading

lostmsu commented Sep 19, 2021

Codecs: customize set of Py<->C# conversions via PyObjectConversions #1022

Codecs: customize set of Py<->C# conversions via PyObjectConversions #1022

Conversation

lostmsu commented Dec 19, 2019 • edited Loading

What does this implement/fix? Explain your changes.

Does this close any currently open issues?

Any other comments?

Checklist

koubaa Dec 21, 2019

Choose a reason for hiding this comment

lostmsu Dec 22, 2019 • edited Loading

Choose a reason for hiding this comment

koubaa Jan 15, 2020

Choose a reason for hiding this comment

koubaa Jan 15, 2020

Choose a reason for hiding this comment

koubaa commented Jan 15, 2020

koubaa Jan 15, 2020

Choose a reason for hiding this comment

lostmsu commented Jan 16, 2020

filmor Jan 17, 2020

Choose a reason for hiding this comment

filmor Jan 17, 2020

Choose a reason for hiding this comment

lostmsu commented Jan 30, 2020

codecov-io commented Feb 1, 2020 • edited Loading

Codecov Report

lostmsu commented Feb 13, 2020

lostmsu commented Feb 26, 2020

amos402 commented Feb 28, 2020

igiona commented Sep 16, 2021

lostmsu commented Sep 16, 2021 • edited Loading

igiona commented Sep 17, 2021

lostmsu commented Sep 17, 2021 • edited Loading

igiona commented Sep 19, 2021 • edited Loading

lostmsu commented Sep 19, 2021

lostmsu commented Dec 19, 2019 •

edited

Loading

lostmsu Dec 22, 2019 •

edited

Loading

codecov-io commented Feb 1, 2020 •

edited

Loading

lostmsu commented Sep 16, 2021 •

edited

Loading

lostmsu commented Sep 17, 2021 •

edited

Loading

igiona commented Sep 19, 2021 •

edited

Loading