Skip to content

Python.exe crashes if __pycache__ directories exist (garbage collection trashed) #481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
testrunner123 opened this issue May 31, 2017 · 16 comments
Labels
Milestone

Comments

@testrunner123
Copy link
Contributor

testrunner123 commented May 31, 2017

Environment

  • Pythonnet version: 2.3.0
  • Python version: 3.5 64-bit
  • Operating System: Windows 7 Pro 64-bit SP1

Details

  • Describe what you were trying to get done.

I am trying to import .NET assemblies and use them together. Unfortunately, python interpreter always crashes if .py files are already precompiled and lay inside pycache directories. I tried breifly python 3.6 and newer pythonnet, but it still crashes. However it doesn't crash with old python 2.7.13 64-bit (need to recompile pythonnet for 2.7).

I used Debugger and see that python garbage collection is trashed. Mostly object->tp_is_gc field is invalid not showing to real function.

Deleteing pycache directories allow test.py to run once.

Please, help in identify the issue and in making the fix.

  • What commands did you run to trigger this issue?

pythonGCCrash.zip

  1. Unpack zip, open Visual Studio 2015 Update 3 Command Prompt in this directory
  2. Need to compile pythonnet for python 3.5 64-bit windows. Copy clr.pyd and Python.Runtime.dll to the directory of run.bat
  3. start run.bat
  4. python.exe crashes after printing "0"
  • If there was a crash, please include the traceback here.
    The crash happens mostly inside visit_decref() function in PyObject_IS_GC(op) macro. It shows either "op" is 0 or "op->tp_is_gc" has a wrong address.
@den-run-ai
Copy link
Contributor

did you try running python with -B flag?

@testrunner123
Copy link
Contributor Author

did you try running python with -B flag?

I've tried this: in 4 of 5 attempts, python interpreter 3.5 64-bit crashes at the exit. No pycache directores exist in this case.

@vmuriart
Copy link
Contributor

vmuriart commented Jun 1, 2017

I haven't seen yoyr sample code, this isn't related to the sub classing bug right? I'm curious if pycache existed when the subclass tests failed

@den-run-ai
Copy link
Contributor

You zip file does not include Axa.* .NET dependencies.

@testrunner123
Copy link
Contributor Author

@vmuriart
I do subclass in sample and I believe it is related. In real assembly I have subclassing and strange effects. If I remove subclassing then it works.

@denfromufa
I do not understand. Well, I am not .NET pro. I compile with run.bat in Visual Studio 2015 Upd3 Developer Prompt and see only:
C:\dev\job3>csc /target:library /debug:full class.cs
Microsoft (R) Visual C# Compiler version 1.3.1.60616
Copyright (C) Microsoft Corporation. All rights reserved.

and class.dll is made somehow.

@testrunner123
Copy link
Contributor Author

@vmuriart
I tested and removing subclassing in sample code or just removing __namespace__ in adapter.py fixes the issue. Do you have an idea what this bug is ? Or may be it is fixed already??

@den-run-ai
Copy link
Contributor

I see that it is self-contained now - Aka namespace is in the same class.cs.

@testrunner123
Copy link
Contributor Author

I nailed bug in src\runtime\typemananger.cs in CreateSubType(): method Runtime.XIncref() is not called, but PyObject is created and added to disposeList. Correct code:

if (0 != Runtime.PyMapping_HasKey(py_dict, assemblyKey.Handle))
{
	var pyAssembly = new PyObject(Runtime.PyDict_GetItem(py_dict, assemblyKey.Handle));
	Runtime.XIncref(pyAssembly.Handle);
	...
}
...
if (0 != Runtime.PyMapping_HasKey(py_dict, namespaceKey.Handle))
{
	var pyNamespace = new PyObject(Runtime.PyDict_GetItem(py_dict, namespaceKey.Handle));
	Runtime.XIncref(pyNamespace.Handle);
...
}

Now it doesn't crash anymore.

Would be not easy to make regression test out of it. I realized that no crash if this code is completely in one .py. It must be spread among several modules to provoke garbage collector crash.

@vmuriart
Copy link
Contributor

vmuriart commented Jun 2, 2017

Nice! The subclass tests might catch this. Even though it's intermittent I run it multiple times each version to check for it.

@testrunner123
Copy link
Contributor Author

If it works in your tests then you can submit this patch to repo.

@den-run-ai
Copy link
Contributor

@testrunner123 would you like to submit a pull request? otherwise someone can do this for you and attribute the commit to you git commit --author="???"

@filmor
Copy link
Member

filmor commented Jun 7, 2017

Hmm, we had quite a few of those. Maybe we should replace the current new PyObject by more explicit constructors like PyObject.FromPtrIncref, PyObject.BorrowFromPtr? I don't really have good names, though.

@den-run-ai
Copy link
Contributor

@filmor i agree about PyObject.FromPtrIncref, when would PyObject.BorrowFromPtr happen?

@den-run-ai den-run-ai added this to the 2.4.0 milestone Jun 14, 2017
@filmor
Copy link
Member

filmor commented Jul 11, 2017

I didn't think much about the names, I'll try to get a PR up with a proposal for explicit constructors.

@filmor filmor modified the milestones: 2.4.0, 2.5.0 Apr 12, 2019
@lostmsu
Copy link
Member

lostmsu commented Apr 23, 2020

@filmor the fix seems to be merged. Can this bug be closed?

@filmor
Copy link
Member

filmor commented Apr 23, 2020

I think so, yes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants