ENH: add mean keyword to std and var #24126

RonaldAJ · 2023-07-05T18:32:58Z

Replaces previous pull request #23889 (ENH: Introduce new functions mean_std and mean_var.) after triage review.

Often when the standard deviation is needed the mean is also needed; the same holds for the variance and the mean. With the current code the mean is then calculated twice, this can be prevented if the functions calculating the variance or the standard deviation can use a precalculated mean. See ticket #23741.

These changes can be largely independent from those for ticket #13199, which suggests reducing memory usage by avoiding storing data-mean in a separate intermediate array.

See also: discussion on the mailing list

mhvk

This seems a good solution! I also like that now one can put in something else than the mean from the same data (which is often useful). But see the important comment about subclasses, plus a lot of more nitpicky ones.

Note that I'm surprised MaskedArray does not work. Did you try again? Could you explain the problem? And is this a new problem, or did the functions not work even before? If they worked before, it would be much nicer to get MaskedArray right too.

numpy/core/fromnumeric.py

numpy/core/tests/test_numeric.py

doc/release/upcoming_changes/24126.change.rst

numpy/core/fromnumeric.py

RonaldAJ · 2023-07-06T10:12:43Z

Note that I'm surprised MaskedArray does not work. Did you try again? Could you explain the problem? And is this a new problem, or did the functions not work even before? If they worked before, it would be much nicer to get MaskedArray right too.

I now added support for "mean" keyword on MaskedArrays.

(numpy#24126 (comment)) The dispatcher returns all arguments that can, in principle, contain something that is an array and hence that, if not a regular ndarray, can allow the function to be dealt with another package (say, dask). Since mean can be an array, it should be added (most similar to where).

RonaldAJ · 2023-07-06T13:02:11Z

@charris The cygwin test failed, but from the reported error I guess it has little to do with my change. Will it run again automagically?

mattip · 2023-07-06T14:16:51Z

I re-ran it manually. If it fails again we can ignore the failure.

mattip

Looks good, @mhvk already did all the review. Just a few nits about the release note and docstrings.

doc/release/upcoming_changes/24126.change.rst

numpy/core/fromnumeric.py

…string.

…ument

numpy/lib/nanfunctions.py

… the example code ugly, alas!

RonaldAJ · 2023-07-06T19:32:47Z

The output of the examples depends on platform and circumstances. Therefore testing based on example code gives errors:

-------------- 2023-07-06T19:09:55.1262830Z 2023-07-06T19:09:55.1263570Z File "build/testenv/lib/python3.9/site-packages/numpy/__init__.py", line ?, in std 2023-07-06T19:09:55.1264100Z Failed example: 2023-07-06T19:09:55.1264890Z print(f'Percentage execution time saved {100*(t2-t1)/t2:.0f}%') 2023-07-06T19:09:55.1265550Z Expected: 2023-07-06T19:09:55.1266030Z Percentage execution time saved 30% 2023-07-06T19:09:55.1266470Z Got: 2023-07-06T19:09:55.1266880Z Percentage execution time saved 32% 2023-07-06T19:09:55.1267080Z 2023-07-06T19:09:55.1267310Z numpy.core.var 2023-07-06T19:09:55.1267770Z -------------- 2023-07-06T19:09:55.1267930Z 2023-07-06T19:09:55.1268440Z File "build/testenv/lib/python3.9/site-packages/numpy/__init__.py", line ?, in var 2023-07-06T19:09:55.1268800Z Failed example: 2023-07-06T19:09:55.1269300Z print(f'Percentage execution time saved {100*(t2-t1)/t2:.0f}%') 2023-07-06T19:09:55.1269630Z Expected: 2023-07-06T19:09:55.1275670Z Percentage execution time saved 32% 2023-07-06T19:09:55.1276120Z Got: 2023-07-06T19:09:55.1276900Z Percentage execution time saved 31%

Is there a method for handling this?

RonaldAJ · 2023-07-06T20:05:21Z

Added: #doctest: +SKIP

mattip · 2023-07-06T20:16:59Z

Is there a method for handling this?

You could do something like

# Using mean saves around 32% of the compute time
>>> time_saved = (t2-t1)/t2
>>> assert 0.3 < time_saved < 0.35

RonaldAJ · 2023-07-07T05:51:50Z

Is there a method for handling this?

You could do something like
# Using mean saves around 32% of the compute time
>>> time_saved = (t2-t1)/t2
>>> assert 0.3 < time_saved < 0.35

#doctest: +SKIP did the trick

mattip · 2023-07-07T05:58:10Z

Thanks @RonaldAJ and @mhvk. This is a nice API addition.

RonaldAJ · 2023-07-07T09:18:08Z

I think that issue #23741 can be closed.

RonaldAJ added 2 commits July 5, 2023 18:23

Add mean keyword to std and var functions.

5d07178

Add releae note for mean keyword to std and var functions.

3ba2ccc

github-actions bot added the 01 - Enhancement label Jul 5, 2023

RonaldAJ added 4 commits July 5, 2023 20:35

Update release note with PR number

d52c0b7

Address lint issue.

3c9e4d3

Align nan signatures with new signatures.

f78e793

Address lint issue.

3bb28dd

mhvk reviewed Jul 5, 2023

View reviewed changes

RonaldAJ added 6 commits July 6, 2023 10:09

Correct version numbers on keywords.

13161f1

Put backticks on keyword argument in documentation string.

d93630d

Cleanuup assert statements in tests

6e966a1

Move comparison of in and out arrays closer to the function call.

fe32fd1

Remove clutter from example code in release note.

a15af41

Add test for nanstd and fix error in nanvar

1a10e70

RonaldAJ added 3 commits July 6, 2023 12:13

haqndle "mean" keyword for var and std on MaskedArrays.

ef7484a

Address lint issues.

ccc5258

mattip reviewed Jul 6, 2023

View reviewed changes

doc/release/upcoming_changes/24126.change.rst Outdated Show resolved Hide resolved

numpy/core/fromnumeric.py Outdated Show resolved Hide resolved

numpy/core/fromnumeric.py Outdated Show resolved Hide resolved

numpy/core/fromnumeric.py Outdated Show resolved Hide resolved

RonaldAJ added 4 commits July 6, 2023 18:20

Move and adjust example from release note to doc-strings. Reflow doc-…

bbdeed4

…string.

Improve doc-string. Shorter sentences and add type and label mean arg…

201034a

…ument

Remove some of these pesky trailing white spaces

71998c8

Make extra white lines more consistent.

0f553ff

mattip reviewed Jul 6, 2023

View reviewed changes

numpy/lib/nanfunctions.py Show resolved Hide resolved

RonaldAJ added 4 commits July 6, 2023 20:07

Make sure code examples execute without Jupyter magic.

aa861b3

Fold lines to pass linter.

020257a

Update doc-string nanstd and nanvar.

f8d8399

Try to satisfy linter and apple requirements at the same time. Making…

4f5cc4c

… the example code ugly, alas!

Make doctest skip resource dependent output

b891190

mattip merged commit ab2178b into numpy:main Jul 7, 2023

seberg added the 62 - Python API Changes or additions to the Python API. Mailing list should usually be notified. label Jul 7, 2023

akx mentioned this pull request Nov 25, 2024

DOC: Correct version-added for mean arg for nanvar and nanstd #27839

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add mean keyword to std and var #24126

ENH: add mean keyword to std and var #24126

RonaldAJ commented Jul 5, 2023

mhvk left a comment

RonaldAJ commented Jul 6, 2023

RonaldAJ commented Jul 6, 2023

mattip commented Jul 6, 2023

mattip left a comment

RonaldAJ commented Jul 6, 2023 •

edited

Loading

RonaldAJ commented Jul 6, 2023

mattip commented Jul 6, 2023

RonaldAJ commented Jul 7, 2023

mattip commented Jul 7, 2023

RonaldAJ commented Jul 7, 2023 •

edited

Loading

ENH: add mean keyword to std and var #24126

ENH: add mean keyword to std and var #24126

Conversation

RonaldAJ commented Jul 5, 2023

mhvk left a comment

Choose a reason for hiding this comment

RonaldAJ commented Jul 6, 2023

RonaldAJ commented Jul 6, 2023

mattip commented Jul 6, 2023

mattip left a comment

Choose a reason for hiding this comment

RonaldAJ commented Jul 6, 2023 • edited Loading

RonaldAJ commented Jul 6, 2023

mattip commented Jul 6, 2023

RonaldAJ commented Jul 7, 2023

mattip commented Jul 7, 2023

RonaldAJ commented Jul 7, 2023 • edited Loading

RonaldAJ commented Jul 6, 2023 •

edited

Loading

RonaldAJ commented Jul 7, 2023 •

edited

Loading