ENH: add support for python3.6 memory tracing #8885
Conversation
5c626c7 to
87c075d
Compare
|
@juliantaylor Could you add a release note? |
|
bigger question is whether we want to merge use of a private api? |
|
OK, let's put this off to 1.14. At this point I'm just waiting on |
87c075d to
21b8ca8
Compare
|
I don't think this is particularly risky. We can also put a version == 3.6 check that we update each release. |
|
"#define NPY_TRACE_DOMAIN 389047" why 389047 and not 0? I added a domain parameter because @njsmith said that numpy may allocate memory in the GPU, and CPU and GPU address spaces can overlap: About the version: CPython 3.7 will likely rename the track/untrack functions to make them public, but it will be trivial to update your patch for this change. For CPython 3.6: I'm the author and the functions and I don't plan to modify or remove them in a stable release. |
|
My though was to allow easy filtering of all allocations that numpy does. numpy allocations are usually large while python (domain 0) are loads of small things. |
|
tracemalloc allows you to filter by filename: Domains is something new in Python 3.6, I didn't use it to track a memory leak. Not all tracemalloc support domains. For example, tracemalloc.get_traced_memory() counts traces of all the domain. It's up to you to use a different domain. |
|
I find using domains for this is more elegant than filename filters. It doesn't prevent you from using them either as long as traceutils doesn't filter out domains unless you ask it to. |
|
Would it be feasible to make the domain configurable? Maybe using an
environment variable?
The domain must not change after the first call to track(). Maybe add
a function to change the domain, but raise an exception in that
function is track() was already called?
|
|
Sure you could make it configurable. Not sure if its worth the effort though. Why would you want to configure it? |
|
Sure you could make it configurable. Not sure if its worth the effort though. Why would you want to configure it?
Mostly to use domain=0. Well, maybe it's overkill. It was just a question :-)
|
|
Would using domain 0 for numpy bring any advantages? |
ab28cda to
181b309
Compare
|
I think this should still be in 1.13, it is pretty simple and it will be a public api in python3.7 so we can pretty safely use it now. |
|
@juliantaylor Needs a release note. |
|
@juliantaylor Could you update the release notes? |
Python 3.6 added a private API for tracing arbitrary memory allocations so the tracemalloc module [0] can be used with numpy. closes numpygh-4663 [0] https://docs.python.org/3/library/tracemalloc.html
add numpy.distutils and tracemalloc changes and fix some issues. [ci skip]
181b309 to
3ba10cf
Compare
|
Thanks Julian. |
Python 3.6 added a private API for tracing arbitrary memory allocations
so the tracemalloc module [0] can be used with numpy.
closes gh-4663
As it is a private api it might not be mergeable (though its not the first private api we are using), but people may want to play around with it now.
[0] https://docs.python.org/3/library/tracemalloc.html