Skip to content

Add soft shutdown #958

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 163 commits into from
Oct 8, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
163 commits
Select commit Hold shift + click to select a range
d1928dc
Drop LoadLibrary dependency
amos402 Jun 13, 2019
f000e08
Explain for getting _PyObject_NextNotImplemented
amos402 Jun 13, 2019
f882400
Update changelog and authors
amos402 Jun 13, 2019
b7715ee
Merge branch 'master' into drop-dlopen
filmor Jun 20, 2019
41ac665
Merge remote-tracking branch 'remotes/upstream/master' into drop-dlopen
amos402 Jul 1, 2019
c6dae9e
Merge branch 'master' into drop-dlopen
amos402 Jul 5, 2019
4e19a4f
Add soft shutdown
amos402 Sep 17, 2019
657452e
* Fix refcnt error
amos402 Sep 19, 2019
91f64b9
Fixed leaking of tp_name
amos402 Sep 20, 2019
b07b844
* Reset type slots
amos402 Sep 20, 2019
7db724e
Clear ExtensionType
amos402 Sep 20, 2019
2a88be4
Del SlotsHolder from tp_dict when shutting down
amos402 Sep 20, 2019
2940973
Fix refcnt error of qualname
amos402 Sep 21, 2019
d108913
Since ClassBase not override tp_init for BaseException, repr(Exceptio…
amos402 Sep 22, 2019
cc9e7e5
* Use environment variable `PYTHONNET_SOFT_SHUTDOWN` to enable soft s…
amos402 Sep 22, 2019
1cb8e8c
Fixed deadlock on finalizer after shutdown
amos402 Oct 13, 2019
653a263
Merge branch 'master' into drop-dlopen
amos402 Dec 1, 2019
b3e889b
* Load PyModuleType without LibraryLoader
amos402 Dec 1, 2019
04d6dfb
Merge remote-tracking branch 'remotes/upstream/master' into drop-dlop…
amos402 Dec 14, 2019
5150e61
Prevent exception override
amos402 Dec 14, 2019
65e209e
Rollback symbol loading for __Internal
amos402 Dec 15, 2019
cad95da
Merge remote-tracking branch 'master' into soft-shutdown
amos402 Dec 20, 2019
0dee5da
Free GC handle for all subclass of ExtensionType
amos402 Dec 20, 2019
00a0b32
Specific exception types
amos402 Dec 20, 2019
77da6df
Record mp_length slot
amos402 Dec 23, 2019
2039e69
* Fix deadlock on domain unload
amos402 Dec 23, 2019
fe5050d
Fix refcnt error
amos402 Dec 26, 2019
593fb00
Reset a tuple for tp_bases
amos402 Dec 26, 2019
1ce83fc
Merge branch 'decref-types' into soft-shutdown
amos402 Jan 17, 2020
fba616a
Fix refcnt error
amos402 Jan 17, 2020
bdc0f72
* Use subtype slots instead JIT code
amos402 Jan 17, 2020
49d98e8
Clear tp_dict of ModuleObject
amos402 Jan 18, 2020
433d0f6
Manipulate SlotsHolder manually instead of Capsule mechanism
amos402 Jan 18, 2020
9b4864b
Merge branch 'master' into soft-shutdown
filmor Jan 22, 2020
1b466df
* Drop NativeCodePage dependency
amos402 Jan 23, 2020
49130c4
Remove unused code
amos402 Jan 23, 2020
76ba510
Add tp_clear for constructorbinding
amos402 Feb 1, 2020
1ff21ac
tp_clear for EventBinding
amos402 Feb 3, 2020
0b01378
All base type corrected, remove unnecessary slot set
amos402 Feb 4, 2020
631bb43
Merge branch 'master' into soft-shutdown
amos402 Feb 5, 2020
bf3d9f8
Add basic `reload` shutdown mode
amos402 Feb 5, 2020
da97502
Data synchronization for PyScopeTest.TestThread
amos402 Feb 5, 2020
e8b3160
Disable `ShutdownMode.Reload` on `NETSTANDARD`
amos402 Feb 6, 2020
80d4fa0
* Serialize CLRObject
amos402 Feb 9, 2020
9874cd1
Add ShutdownMode.Default refer to normal mode if no EnvironmentVariab…
amos402 Feb 9, 2020
8da561b
Not generating the debug info for generated modules
amos402 Feb 9, 2020
9499c64
`TestDomainReload` use itself assembly instead of dynamic creation
amos402 Feb 10, 2020
431d644
Merge branch 'master' into drop-dlopen
amos402 Feb 12, 2020
2b84394
* Fix refcnt error
amos402 Feb 12, 2020
5ade069
Validate return value
amos402 Feb 12, 2020
8dabed7
Merge branch 'drop-dlopen' into soft-shutdown
amos402 Feb 13, 2020
f4bb77a
* API for getting the default shutdown mode
amos402 Feb 13, 2020
670bd74
In domain test, use soft mode if default mode is normal
amos402 Feb 13, 2020
3cb56f1
Avoid Domain tests influence other tests when the default is not Soft…
amos402 Feb 13, 2020
3c9a83c
Skip non-serializable objects
amos402 Feb 13, 2020
992c469
Merge branch 'soft-shutdown-reload' into soft-shutdown
amos402 Feb 13, 2020
924b217
Merge branch 'master' into soft-shutdown
amos402 Feb 13, 2020
df84e29
* Reset Exceptions to IntPtr.Zero
amos402 Feb 17, 2020
8b51621
Split RuntimeData into separate file
amos402 Feb 17, 2020
97c8c2a
Extract InitPyMembers method
amos402 Feb 21, 2020
39f47c8
* Classify runtime data
amos402 Feb 24, 2020
aa63f0b
* Stash for ImportHook
amos402 Feb 27, 2020
5b2f3d4
Merge branch 'master' into soft-shutdown
amos402 Feb 27, 2020
35cbe55
Apply Reference type usage
amos402 Feb 27, 2020
f23cae6
* Manipulate refcnt in Push/Pop objects
amos402 Mar 2, 2020
183f9d8
Load cache of ModuleObject after reload
amos402 Mar 8, 2020
9d57a82
Add temp tests reference by https://github.com/pythonnet/pythonnet/pu…
amos402 Mar 8, 2020
08fad26
* Fix refcnt error of MethodBinding
amos402 Mar 9, 2020
bcfdcc7
Test for class object on crossed domain
amos402 Mar 10, 2020
66ab719
Multi times for running cross dispose
amos402 Mar 10, 2020
1428af3
Merge branch 'master' into soft-shutdown
amos402 Mar 10, 2020
8e3c028
Apply Reference type usage
amos402 Mar 10, 2020
b387e9e
Merge branch 'master' into soft-shutdown
amos402 Mar 13, 2020
f85999e
Merge branch 'master' into soft-shutdown
amos402 Jun 16, 2020
cb65af3
Manually merge - remove redundant code
amos402 Jun 16, 2020
498fc8c
Manually merge - ManagedDataOffsets
amos402 Jun 16, 2020
cc2219e
Manually merge - capi prototype
amos402 Jun 16, 2020
8c8d66e
* Move fields of ManagedDataOffsets into nested type
amos402 Jun 24, 2020
4f00165
Avoid mess up the debug info of runtime module by emit IL code
amos402 Jun 24, 2020
39e20e3
Merge branch 'master' into soft-shutdown
amos402 Jun 24, 2020
da7c150
* Fix syntax error
amos402 Jun 24, 2020
02b1ada
Merge branch 'master' into soft-shutdown
amos402 Jul 2, 2020
a8840b2
Drop Python 2.7 on CI
amos402 Jul 2, 2020
dec7a74
Fix refcnt error
amos402 Jul 11, 2020
e877b33
Get platform information without import `platform`
amos402 Jul 11, 2020
6d738bf
Run callbacks registered by `atexit` at Shutdown on soft-shutdown mode
amos402 Jul 11, 2020
ff5edc3
Use named shutdown-mode value by environment value instead of simple …
amos402 Jul 11, 2020
5ac75ba
Remove dependency on importing `platform`. #891
amos402 Jul 15, 2020
65cb22e
Don't call exit functions on soft or reload shutdown
BadSingleton Aug 14, 2020
73865d4
Adding to AUTHORS.MD
BadSingleton Aug 19, 2020
1a75f51
Code review fixes
BadSingleton Aug 19, 2020
9b6d140
Merge pull request #1 from Unity-Technologies/reload-shutdown-fix
amos402 Aug 20, 2020
d9d5562
Remove unused code
BadSingleton Aug 19, 2020
9b62a61
Fixes some typos in TestDomainReload
BadSingleton Aug 19, 2020
4f0420e
Adds missing assert
BadSingleton Aug 19, 2020
4ab9f1c
Adds code comments to CrossDomainObject test
BadSingleton Aug 19, 2020
32bcb3a
Adds documentation to TestDomainReload.TestClassReference
BadSingleton Aug 19, 2020
0077ea8
Adds numbering to domain names in TestDomainReload
BadSingleton Aug 19, 2020
38ea0b6
Use Console.WriteLine formatting overload
BadSingleton Aug 19, 2020
06a656e
Inline called-once methods in TestDomainReload
BadSingleton Aug 19, 2020
09f8281
Simplify PythonRunner.RunPython
BadSingleton Aug 20, 2020
7ec9a6c
Release the GIL
BadSingleton Aug 20, 2020
802a43a
BorrowReference instead of increasing the refcount
BadSingleton Aug 20, 2020
0fdf969
Add IsTypeObject to Managed type
BadSingleton Aug 20, 2020
3a8c72d
Rename PyObjectSerializeContext
BadSingleton Aug 20, 2020
b52bc01
Rename StashPush/Pop methods
BadSingleton Aug 20, 2020
bfbf2c3
Rename ClassManager.RemoveClasses
BadSingleton Aug 20, 2020
883c4ce
Rename ClassManager.OnVisit
BadSingleton Aug 20, 2020
7e0d56d
fixup! Inline called-once methods in TestDomainReload
BadSingleton Aug 21, 2020
82034dc
Change refcount logic in ImportHook.Shutdown
BadSingleton Aug 21, 2020
4ba50a7
Remove TODOS
BadSingleton Aug 21, 2020
1ecdce8
Incref a valid pointer
BadSingleton Aug 21, 2020
b35f441
Use Py_CLEAR
BadSingleton Aug 21, 2020
ce8ee90
Fixes GIL grabbing during init and shutdown
BadSingleton Aug 21, 2020
4c4bcb0
Add a parameter to specify the shutdown mode.
BadSingleton Aug 26, 2020
f575bd3
Add typeoffset.cs
BadSingleton Aug 26, 2020
d1799aa
Extract utility method in TestDomainReload
BadSingleton Aug 26, 2020
d2408b9
Add cleanup to ClassInfo
BadSingleton Aug 26, 2020
fd2b662
Update comment of `ClassManager.RemoveClasses`
amos402 Aug 27, 2020
2b7bcac
Fix typo errors
amos402 Aug 27, 2020
0af3504
Merge remote-tracking branch 'amos/soft-shutdown' into soft-shutdown-…
BadSingleton Aug 27, 2020
ba1df6e
fixup! Add cleanup to ClassInfo
BadSingleton Aug 27, 2020
639ba1f
Add a new PyList constructor
BadSingleton Aug 27, 2020
7e5ab52
Revert previous commit "Add cleaunp to class info"
BadSingleton Aug 27, 2020
8075f48
Add corrections to the GIL acquiring/releasing
BadSingleton Aug 27, 2020
eb8cb8a
Factor out ExtensionType's Gc setup mechanic
BadSingleton Aug 27, 2020
308f0f2
Revert "Release the GIL"
BadSingleton Aug 28, 2020
1ae0bfe
fixup! Incref a valid pointer
BadSingleton Aug 28, 2020
07aefe6
Revert "Add typeoffset.cs"
BadSingleton Aug 28, 2020
5387b05
Merge remote-tracking branch 'remotes/upstream/master' into soft-shut…
amos402 Aug 31, 2020
b203674
Revert local project setting
amos402 Aug 31, 2020
19d1379
A better fix for releasing a re-acquiring the GIL
BadSingleton Sep 2, 2020
25a4064
Add validation for downgrading the shutdown mode
BadSingleton Sep 2, 2020
1272b89
Merge remote-tracking branch 'amos/soft-shutdown' into soft-shutdown-…
BadSingleton Sep 2, 2020
8c133e3
Fixes for the merge
BadSingleton Sep 2, 2020
d9b21a5
Revert "Change refcount logic in ImportHook.Shutdown"
BadSingleton Sep 2, 2020
c8dee53
Document refcount increase
BadSingleton Sep 2, 2020
598cb77
Disambiguate Initialization
BadSingleton Sep 3, 2020
83e8dd5
Merge pull request #2 from Unity-Technologies/soft-shutdown-review-co…
amos402 Sep 5, 2020
6db3181
Revert local changes
amos402 Sep 9, 2020
e38a363
Remove unused code
amos402 Sep 9, 2020
b409a89
Check serializability of the whole type hierarchy
BadSingleton Sep 8, 2020
f5c24b0
Release the GIL on shutdown only if it can be released
BadSingleton Sep 8, 2020
0d6c645
Call `WaitForPendingFinalizers` instead
BadSingleton Sep 8, 2020
fa89b48
Revert local changes
amos402 Sep 9, 2020
80a7644
Remove unused code
amos402 Sep 9, 2020
12c0206
Merge pull request #3 from Unity-Technologies/soft-shutdown-review-co…
amos402 Sep 11, 2020
6a3cfc8
Adds an unchecked version to get a BorrowedReference pointer
BadSingleton Sep 15, 2020
98da1fa
Remove compile-time check on NETSTANDARD
BadSingleton Sep 15, 2020
d7d44e8
fixup! Remove compile-time check on NETSTANDARD
BadSingleton Sep 15, 2020
6aa75c5
Factor out the clearing of clr_data
Sep 16, 2020
2343f89
Partially Revert "Adds an unchecked version to get a BorrowedReferenc…
Sep 16, 2020
cc6b8e4
More changes to use BorrowedReference and NewReference
BadSingleton Sep 24, 2020
d5fcfa4
fixup! More changes to use BorrowedReference and NewReference
BadSingleton Sep 24, 2020
fa47957
fixup! fixup! More changes to use BorrowedReference and NewReference
BadSingleton Sep 25, 2020
ff956e4
Remove unused method
BadSingleton Sep 25, 2020
c7b134c
Merge pull request #4 from Unity-Technologies/soft-shutdown-review-co…
amos402 Sep 29, 2020
97e61a5
Merge branch 'master' into soft-shutdown
amos402 Sep 30, 2020
0b9d2c1
Wait for full GC Complete
amos402 Sep 30, 2020
178cbc8
Release atexit manually
amos402 Sep 30, 2020
3a17f36
* Remove `fromPython`
amos402 Sep 30, 2020
3203457
Merge branch 'master' into soft-shutdown
amos402 Oct 8, 2020
8d00e4c
Merge branch 'master' into soft-shutdown
amos402 Oct 8, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Manually merge - ManagedDataOffsets
  • Loading branch information
amos402 committed Jun 16, 2020
commit 498fc8cd89ad13623f56670961474a35deb7c51a
42 changes: 29 additions & 13 deletions src/runtime/interop.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
using System.Runtime.InteropServices;
using System.Reflection;
using System.Text;
using System.Collections.Generic;
using System.Linq;

namespace Python.Runtime
Expand Down Expand Up @@ -70,15 +69,20 @@ public ModulePropertyAttribute()
}
}

internal static partial class ManagedDataOffsets
internal static partial class TypeOffset
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't this be in typeoffset.cs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could, but this change is getting big. I'd prefer doing it after this branch has been merged

{
static class Helper
{
public static int magic;
public static readonly Dictionary<string, int> NameMapping = new Dictionary<string, int>();
}
public static int magic() => ManagedDataOffsets.Magic;
}

static TypeOffset()
internal static class ManagedDataOffsets
{
public static int Magic { get; private set; }
public static readonly Dictionary<string, int> NameMapping = new Dictionary<string, int>();

public static readonly int ob_data;
public static readonly int ob_dict;

static ManagedDataOffsets()
{
Type type = typeof(TypeOffset);
FieldInfo[] fields = type.GetFields();
Expand All @@ -88,18 +92,30 @@ static TypeOffset()
int offset = i * size;
FieldInfo fi = fields[i];
fi.SetValue(null, offset);
Helper.NameMapping[fi.Name] = offset;
NameMapping[fi.Name] = offset;
}
// XXX: Use the members after PyHeapTypeObject as magic slot
Helper.magic = members;
Magic = TypeOffset.members;
}

public static int magic() => Helper.magic;

public static int GetSlotOffset(string name)
{
return Helper.NameMapping[name];
return NameMapping[name];
}

private static int BaseOffset(IntPtr type)
{
Debug.Assert(type != IntPtr.Zero);
int typeSize = Marshal.ReadInt32(type, TypeOffset.tp_basicsize);
Debug.Assert(typeSize > 0 && typeSize <= ExceptionOffset.Size());
return typeSize;
}

public static int DataOffset(IntPtr type)
{
return BaseOffset(type) + ob_data;
}

public static int DictOffset(IntPtr type)
{
return BaseOffset(type) + ob_dict;
Expand Down
4 changes: 2 additions & 2 deletions src/runtime/managedtype.cs
Original file line number Diff line number Diff line change
Expand Up @@ -226,12 +226,12 @@ protected static void ClearObjectDict(IntPtr ob)

protected static IntPtr GetObjectDict(IntPtr ob)
{
return Marshal.ReadIntPtr(ob, ObjectOffset.DictOffset(ob));
return Marshal.ReadIntPtr(ob, ObjectOffset.TypeDictOffset(ob));
}

protected static void SetObjectDict(IntPtr ob, IntPtr value)
{
Marshal.WriteIntPtr(ob, ObjectOffset.DictOffset(ob), value);
Marshal.WriteIntPtr(ob, ObjectOffset.TypeDictOffset(ob), value);
}
}
}
6 changes: 3 additions & 3 deletions src/runtime/typemanager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -638,7 +638,7 @@ internal static void InitializeSlots(IntPtr type, Type impl, SlotsHolder slotsHo
{
continue;
}
var offset = TypeOffset.GetSlotOffset(slot);
var offset = ManagedDataOffsets.GetSlotOffset(slot);
Marshal.WriteIntPtr(type, offset, SlotsHolder.GetDefaultSlot(offset));
}
}
Expand All @@ -656,7 +656,7 @@ internal static void InitializeSlots(IntPtr type, Type impl, SlotsHolder slotsHo
/// <param name="canOverride">Can override the slot when it existed</param>
static void InitializeSlot(IntPtr type, IntPtr slot, string name, bool canOverride = true)
{
var offset = TypeOffset.GetSlotOffset(name);
var offset = ManagedDataOffsets.GetSlotOffset(name);
if (!canOverride && Marshal.ReadIntPtr(type, offset) != IntPtr.Zero)
{
return;
Expand Down Expand Up @@ -693,7 +693,7 @@ static void InitializeSlot(IntPtr type, int slotOffset, MethodInfo method, Slots

static bool IsSlotSet(IntPtr type, string name)
{
int offset = TypeOffset.GetSlotOffset(name);
int offset = ManagedDataOffsets.GetSlotOffset(name);
return Marshal.ReadIntPtr(type, offset) != IntPtr.Zero;
}

Expand Down