API: deprecate undocumented functions #24154

mtsokol · 2023-07-10T13:34:10Z

Here I share a PR with a next batch of deprecated functions from #12385. In this PR deprecated functions are:

recfromcsv
recfromtxt
disp
get_array_wrap
maximum_sctype
deprecate
deprecate_with_doc

(they aren't used except for dedicated tests so dropping them for good should be just removing them)

[EDIT]
Also this PR un-deprecate byte_bounds as agreed due to downstream impact.

seberg · 2023-07-10T13:41:43Z

Can we just avoid @deprecate (we can deprecate it also, or just remove it TBH, I don't see why downstream needs our helper, they can just vendor it).
It doesn't provide a nice message or docs update. I am not even sure it provides the right stacklevel (which leads to issues like gh-24015, although there the stacklevel is just wrong stacklevel).

mtsokol · 2023-07-10T13:49:47Z

Can we just avoid @deprecate (we can deprecate it also, or just remove it TBH, I don't see why downstream needs our helper, they can just vendor it). It doesn't provide a nice message or docs update. I am not even sure it provides the right stacklevel (which leads to issues like gh-24015, although there the stacklevel is just wrong stacklevel).

Hi @seberg,
Sure! I can remove it, just to explain - I used @deprecate in the previous PR #23830.
Then, as I understand, deprecation should happen with a simple

warnings.warn("...", DeprecationWarning)

at the top of deprecated function, is it correct? Or is there another approach?

seberg · 2023-07-10T13:56:32Z

For these functions yes (they don't have a array_function_dispatch decoration). You also need a stacklevel=2 if the function is directly called by the user (as is the case). We normally put a # Deprecated 2023-07-10, NumPy 2.0 above it and a (deprecated in NumPy 2.0) in the warning itself so that users have an idea for how new the warning is it at least.

ngoldbaum · 2023-07-10T16:02:34Z

It would be good to suggest migration paths for users of these functions in the deprecation. If no good migration path is available, we may want to reconsider deprecating. I would search for downstream usages to get a feeling for how functions you'd like to deprecate are used in the real world. For example, the nilearn library seems to have some usages of recfromcsv in their tests and documentation.

mtsokol · 2023-07-10T17:34:53Z

It would be good to suggest migration paths for users of these functions in the deprecation. If no good migration path is available, we may want to reconsider deprecating. I would search for downstream usages to get a feeling for how functions you'd like to deprecate are used in the real world. For example, the nilearn library seems to have some usages of recfromcsv in their tests and documentation.

Hi @ngoldbaum,
Sure! I see that previous deprecation PR had a downstream impact #23830 (comment) so I think it will be helpful to also provide substitutes for deprecated functions in the deprecation message.
Let me maybe prepare a summary of all deprecates/plans from #12385 and NEP 52 in a single table/memo with proposed replacements and downstream impacts and then we can move forward with it. Does it make sense?

rgommers · 2023-07-10T17:57:03Z

Let me maybe prepare a summary of all deprecates/plans from #12385 and NEP 52 in a single table/memo with proposed replacements and downstream impacts and then we can move forward with it. Does it make sense?

I like that plan - much nicer than having separate release note snippets for each deprecation. Perhaps it will end up looking like this page: https://numpy.org/devdocs/reference/array_api.html#table-of-differences-between-numpy-array-api-and-numpy?

If there's a simple one-for-one replacement, it's good to have that in the deprecation message too.

mtsokol · 2023-07-11T20:35:06Z

For these functions yes (they don't have a array_function_dispatch decoration). You also need a stacklevel=2 if the function is directly called by the user (as is the case). We normally put a # Deprecated 2023-07-10, NumPy 2.0 above it and a (deprecated in NumPy 2.0) in the warning itself so that users have an idea for how new the warning is it at least.

Hi @seberg! I deprecated deprecate and deprecate_with_doc and moved to manual warning with DeprecationWarning.

It will itself be deprecated in numpy soon, xref numpy/numpy#24154

rgommers · 2023-07-11T21:35:18Z

The test failures are real, all due to the same leftover usage of np.deprecate in a test it looks like.

ngoldbaum

While looking at the current devdocs I noticed that the deprecate decorator is adding a note to the docstrings of the deprecated functions that they are deprecated. Maybe it would be a good idea to manually add a note to the docstrings of deprecated functions that they were deprecated in NumPy 2.0?

numpy/core/arrayprint.py

ngoldbaum · 2023-07-12T15:47:01Z

numpy/core/numerictypes.py

+
+    # Deprecated in NumPy 2.0, 2023-07-11
+    warnings.warn(
+        "`maximum_sctype` is deprecated. "


I don't have a strong opinion here, but this one seems somewhat useful. It also got docs in #13054, so it's not really undocumented. You could replace it with something like:

def largest_of_kind(kind): return max(np.sctypes[kind], key=lambda x: np.dtype(x).itemsize)

But that seems like unnecessary downstream churn given that this function hasn't been a maintenance burden.

It doesn't seem all that useful to me - the linked issue points at https://stackoverflow.com/questions/48410604/find-maximum-integer-type-at-runtime-in-numpy/48412971#48412971, but the original question there doesn't really make much sense, I can't think of any reason why you'd want your code to dynamically determine the largest unsigned integer like that (it's always uint64) nor why you would want the answer to change in the unlikely case NumPy adds or remove a dtype.

I don't think any of the sctype functions have enough value to have them stay, as touched upon in gh-17325 and explicitly mentioned in https://numpy.org/neps/nep-0052-python-api-cleanup.html#cleaning-up-the-main-namespace (no one disagreed with that I believe).

Sure, not advocating to keep it, just trying to think about how you'd replace it if you were using it.

This one does seem to have downstream usages, so I think it's worth thinking about what to suggest as a replacement.

I'd suggest not doing that. None of that code looks reasonable. E.g., in the AstroPy one it seems an elaborate way to spell np.uint64 only, so suggesting any direct replacement is worse than just having the author think what their code was supposed to do and then clean it up.

Same for most of those other code examples, a good fraction of which use np.float which no longer exists.

How about something like this?

Suggested change

"`maximum_sctype` is deprecated. "

"`maximum_sctype` is deprecated. Use an explicit dtype like int64 or float64 instead."

That works, thanks. At least it should cover all the example usages I've looked at.

Sure! I included recommended message.

numpy/lib/utils.py

It will itself be deprecated in numpy soon, xref numpy/numpy#24154

mtsokol · 2023-07-30T09:00:04Z

While looking at the current devdocs I noticed that the deprecate decorator is adding a note to the docstrings of the deprecated functions that they are deprecated. Maybe it would be a good idea to manually add a note to the docstrings of deprecated functions that they were deprecated in NumPy 2.0?

Hi @ngoldbaum,
deprecate adds <func_name> is deprecated! Use <replacement> instead., I can add this preamble to every function's stringdoc that I deprecated - should I add it?

seberg · 2023-07-30T09:40:32Z

The nicest thing to to should be adding:

.. deprecated:: 2.0
   Additional guidance on replacement if you have it.  (can be left empty)

I guess after the initial sentence probably, although you can see how it is used otherwise (not sure how often it is used for sentences).

That will render as a nice box.

mtsokol · 2023-07-31T13:44:32Z

Hi! This PR is ready for another round of review (Four CI checks failed due to Do you want to continue? [Y/n] Abort. statement on installation).

ngoldbaum

This looks great and should be a good template for the work necessary for future python API deprecations.

ngoldbaum · 2023-07-31T16:02:22Z

I opened #24303 which should hopefully avoid the test failures you're seeing on Cirrus.

rgommers

LGTM too!

ngoldbaum · 2023-07-31T16:59:32Z

Thanks @mtsokol!

API: deprecate undocumented functions

ab17394

github-actions bot added the 30 - API label Jul 10, 2023

API: apply review comments

017cef3

mtsokol force-pushed the deprecate-undocumented-functions branch from f53ae1e to 017cef3 Compare July 11, 2023 20:25

rgommers added a commit to rgommers/scipy that referenced this pull request Jul 11, 2023

MAINT: remove usages of @np.deprecate

7cdb74f

It will itself be deprecated in numpy soon, xref numpy/numpy#24154

API: remove type tests for deprecate and deprecate_with_doc

be9020d

rgommers mentioned this pull request Jul 12, 2023

API: deprecate compat and selected lib utils #23830

Merged

ngoldbaum reviewed Jul 12, 2023

View reviewed changes

alugowski pushed a commit to alugowski/scipy that referenced this pull request Jul 16, 2023

MAINT: remove usages of @np.deprecate

a590e57

It will itself be deprecated in numpy soon, xref numpy/numpy#24154

mtsokol and others added 2 commits July 30, 2023 10:34

API: apply review comments

703f143

Merge branch 'main' into deprecate-undocumented-functions

bec2b34

API: Add docstring deprecations

80b4fc7

ngoldbaum mentioned this pull request Jul 31, 2023

CI: apt update before apt install on cirrus #24303

Merged

ngoldbaum approved these changes Jul 31, 2023

View reviewed changes

rgommers approved these changes Jul 31, 2023

View reviewed changes

ngoldbaum merged commit 684947a into numpy:main Jul 31, 2023

mtsokol mentioned this pull request Aug 1, 2023

ENH: Overhaul of NumPy main namespace [NEP 52] #24306

Closed

neutrinoceros mentioned this pull request Aug 2, 2023

MNT: avoid numpy.maximum_sctype (deprecated in numpy 2.0) astropy/astropy#15116

Merged

braingram mentioned this pull request Aug 2, 2023

fix Converter type indexing for Converters not supported by Extension asdf-format/asdf#1594

Merged

mtsokol deleted the deprecate-undocumented-functions branch August 2, 2023 21:27

mtsokol mentioned this pull request Aug 23, 2023

DOC: Add missing changelogs for NEP 52 PRs #24510

Merged

ngoldbaum added the Numpy 2.0 API Changes label Aug 24, 2023

InessaPawson mentioned this pull request Sep 18, 2023

DOC: fromnumeric.py - list of functions that are missing examples – tracking issue #22266

Closed

16 tasks

bersbersbers mentioned this pull request Nov 6, 2023

DOC: Improve migration instructions for maximum_sctype #25074

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API: deprecate undocumented functions #24154

API: deprecate undocumented functions #24154

mtsokol commented Jul 10, 2023 •

edited

Loading

seberg commented Jul 10, 2023

mtsokol commented Jul 10, 2023 •

edited

Loading

seberg commented Jul 10, 2023

ngoldbaum commented Jul 10, 2023 •

edited

Loading

mtsokol commented Jul 10, 2023 •

edited

Loading

rgommers commented Jul 10, 2023

mtsokol commented Jul 11, 2023

rgommers commented Jul 11, 2023

ngoldbaum left a comment

ngoldbaum Jul 12, 2023

rgommers Jul 12, 2023

ngoldbaum Jul 12, 2023

rgommers Jul 12, 2023

ngoldbaum Jul 12, 2023 •

edited

Loading

rgommers Jul 12, 2023

mtsokol Jul 30, 2023

mtsokol commented Jul 30, 2023

seberg commented Jul 30, 2023

mtsokol commented Jul 31, 2023

ngoldbaum left a comment

ngoldbaum commented Jul 31, 2023 •

edited

Loading

rgommers left a comment

ngoldbaum commented Jul 31, 2023

	"`maximum_sctype` is deprecated. "
	"`maximum_sctype` is deprecated. Use an explicit dtype like int64 or float64 instead."

API: deprecate undocumented functions #24154

API: deprecate undocumented functions #24154

Conversation

mtsokol commented Jul 10, 2023 • edited Loading

seberg commented Jul 10, 2023

mtsokol commented Jul 10, 2023 • edited Loading

seberg commented Jul 10, 2023

ngoldbaum commented Jul 10, 2023 • edited Loading

mtsokol commented Jul 10, 2023 • edited Loading

rgommers commented Jul 10, 2023

mtsokol commented Jul 11, 2023

rgommers commented Jul 11, 2023

ngoldbaum left a comment

Choose a reason for hiding this comment

ngoldbaum Jul 12, 2023

Choose a reason for hiding this comment

rgommers Jul 12, 2023

Choose a reason for hiding this comment

ngoldbaum Jul 12, 2023

Choose a reason for hiding this comment

rgommers Jul 12, 2023

Choose a reason for hiding this comment

ngoldbaum Jul 12, 2023 • edited Loading

Choose a reason for hiding this comment

rgommers Jul 12, 2023

Choose a reason for hiding this comment

mtsokol Jul 30, 2023

Choose a reason for hiding this comment

mtsokol commented Jul 30, 2023

seberg commented Jul 30, 2023

mtsokol commented Jul 31, 2023

ngoldbaum left a comment

Choose a reason for hiding this comment

ngoldbaum commented Jul 31, 2023 • edited Loading

rgommers left a comment

Choose a reason for hiding this comment

ngoldbaum commented Jul 31, 2023

mtsokol commented Jul 10, 2023 •

edited

Loading

mtsokol commented Jul 10, 2023 •

edited

Loading

ngoldbaum commented Jul 10, 2023 •

edited

Loading

mtsokol commented Jul 10, 2023 •

edited

Loading

ngoldbaum Jul 12, 2023 •

edited

Loading

ngoldbaum commented Jul 31, 2023 •

edited

Loading