Add a new memleak script that does everything #5360
Conversation
|
Not sure how to milestone this. Since it's just a dev utility it can probably go just about anywhere. Also, I should add since this uses the new |
| garbage_arr[i] = garbage | ||
|
|
||
| print('Average memory consumed per loop: %1.4f bytes\n' % | ||
| (np.sum(rss_arr[starti+1:] - rss_arr[starti:-1]) / float(endi - starti))) |
There was a problem hiding this comment.
This looks like just the sum of the differences, which is the end value minus the start value.
((rss_arr[-1] - rss_arr[starti]) / float(endi - starti))There was a problem hiding this comment.
I suppose that's true. We need a different mechanism, then -- something that will take into account the spikiness of the data. If you select the start and end points incorrectly here you get wildly different results.
|
Mike, in the top subplot it looks like pymalloc is showing small fluctuations near 13M, and rss is showing small fluctuations around 150k--maybe rss units here are 512b or 1k blocks. |
|
Yes -- rss is in different units. I just haven't gone and implemented that (because the units depend on platform, version of platform etc.) In any case, they should be on different scales, because pymalloc (including only allocations from Python interpreter itself) will always be significantly smaller than rss. The important thing is not their relative sizes but the first derivative anyway. |
This replaces our 4 memleak scripts with one that is able to test any backend, with or without plot content, and with or without interactive mode. The calculation of average increase per iteration has been fixed. Before, it assumed the increase was monotonically increasing, when in fact it flucuates quite a bit. Therefore, it now calculates the difference between each pair of results and averages that. Also, the results are stored in pre-allocated Numpy arrays rather than Python lists to avoid including the increasing size of the Python lists in the results.
|
I've updated this so that the average memory increase is calculated based on the peak memory usage rather than an instantaneous reading. This should get around the problem where the reading seems artificially high if it happens to pick a valley as the start end point. This has also been updated to use the |
6f932a1 to
b59627b
Compare
|
I don't think we have to back-port this to any other branch unless we want to go memory hunting on them. |
This replaces our 4 memleak scripts with one that is able to test any
backend, with or without plot content, and with or without interactive
mode.
The calculation of average increase per iteration has been fixed.
Before, it assumed the increase was monotonically increasing, when in
fact it flucuates quite a bit. Therefore, it now calculates the
difference between each pair of results and averages that.
Also, the results are stored in pre-allocated Numpy arrays rather than
Python lists to avoid including the increasing size of the Python lists
in the results.