Skip to content

ENH: Overhaul of NumPy lib namespace [NEP 52] #24507

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
16 tasks done
mtsokol opened this issue Aug 23, 2023 · 9 comments
Closed
16 tasks done

ENH: Overhaul of NumPy lib namespace [NEP 52] #24507

mtsokol opened this issue Aug 23, 2023 · 9 comments
Assignees
@mtsokol
Labels
Numpy 2.0 API Changes

Comments

@mtsokol
Copy link
Member

mtsokol commented Aug 23, 2023 •
edited
Loading

Hi @rgommers @ngoldbaum,

This issue is meant for monitoring and discussing cleanup of np.lib namespace. (Tracking parent issue: #23999)

Main goal is to make each public function, which lives in np.lib, available either from the main namespace np or "local lib" namespace e.g. np.lib.npyio. As a result, there will be only one way to access a specific public function in NumPy.
To achieve it, for each relevant lib submodule, there will be a private file (that starts with _) imported by the main namespace, and a public file that allows to be accessed as a public submodule.

# Example:

np.broadcast_to  # imported from numpy/lib/_stride_tricks.py

np.lib.stride_tricks.normalize_axis_tuple  # imported from numpy/lib/stride_tricks.py

Let's call this mechanism "main or submodule namespace access". (maybe a better name?)

The scope of the milestone:

"ns" stands for "namespace". ☑️ means that this submodule has been refactored.

  • lib PR
    main ns: N/A
    local ns: NumpyVersion, Arrayterator, add_newdoc, add_docstring, tracemalloc_domain
  • array_utils PR
    main ns: N/A
    local ns: byte_bounds, discover_array_parameters, normalize_axis_index, normalize_axis_tuple
  • histograms PR
    main ns: histogram, histogramdd, histogram_bin_edges
    local ns:
  • type_check PR
    main ns: iscomplexobj, isrealobj, imag, iscomplex, isreal, nan_to_num, real, real_if_close, typename, mintypecode, common_type
    local ns:
  • nanfunctions PR
    main ns: nansum, nanmax, nanmin, nanargmax, nanargmin, nanmean, nanmedian, nanpercentile, nanvar, nanstd, nanprod, nancumsum, nancumprod, nanquantile
    local ns:
  • function_base PR
    main ns: select, piecewise, trim_zeros, copy, iterable, percentile, diff, gradient, angle, unwrap, sort_complex, flip, rot90, extract, place, vectorize, asarray_chkfinite, average, bincount, digitize, cov, corrcoef, median, sinc, hamming, hanning, bartlett, blackman, kaiser, i0, meshgrid, delete, insert, append, interp, quantile
    local ns:
  • shape_base PR
    main ns: column_stack, row_stack, dstack, array_split, split, hsplit, vsplit, dsplit, apply_over_axes, expand_dims, apply_along_axis, kron, tile, get_array_wrap, take_along_axis, put_along_axis
    local ns:
  • arraypad PR
    main ns: pad
    local ns:
  • arraysetops PR
    main ns: ediff1d, intersect1d, setxor1d, union1d, setdiff1d, unique, isin
    local ns:
  • twodim_base PR
    main ns: diag, diagflat, eye, fliplr, flipud, tri, triu, tril, vander, histogram2d, mask_indices, tril_indices, tril_indices_from, triu_indices, triu_indices_from
    local ns:
  • ufunclike PR
    main ns: fix, isneginf, isposinf
    local ns:
  • utils PR
    main ns: get_include, info, show_runtime
    local ns:
  • npyio PR
    main ns: savetxt, loadtxt, genfromtxt, load, save, savez, savez_compressed, packbits, unpackbits, fromregex,
    local ns: DataSource, NpzFile, BagObj
  • polynomial PR
    main ns: poly, roots, polyint, polyder, polyadd, polysub, polymul, polydiv, polyval, poly1d, polyfit
    local ns:
  • stride_tricks PR
    main ns: broadcast_to, broadcast_arrays, broadcast_shapes
    local ns: as_strided, sliding_window_view
  • index_tricks PR
    main ns: diag_indices_from, diag_indices, fill_diagonal, ndindex, ndenumerate, ix_, ogrid, mgrid, unravel_index, ravel_multi_index, s_, c_, r_, index_exp
    local ns:

This work should be less disturbing than main namespace cleanup, as I think a function is rarely imported from np.lib if it's already available in np.
@mhvk
Copy link
Contributor

mhvk commented Aug 23, 2023

@mtsokol - be careful - lib.stride_tricks is meant to remain (I didn't check all others, but this is one I use (too?) regularly, and the contents are not all exported; see NEP).

@mtsokol
Copy link
Member Author

mtsokol commented Aug 23, 2023 •
edited
Loading

@mtsokol - be careful - lib.stride_tricks is meant to remain (I didn't check all others, but this is one I use (too?) regularly, and the contents are not all exported; see NEP).

Hi @mhvk! Sure, that's correct.
For example np.lib.stride_tricks.as_strided or normalize_axis_tuple are the ones that are public and available only from the submodule.

The ones from lib.stride_tricks that are being exported and will be accessible only from main namespace are broadcast_to, broadcast_arrays and broadcast_shapes.

@mhvk
Copy link
Contributor

mhvk commented Aug 23, 2023

Yes, as_strided remains a favourite... There's also sliding_window_view (but normalize_axis_tuple comes from elsewhere).

@mtsokol mtsokol mentioned this issue Aug 23, 2023
Merged
@rgommers
Copy link
Member

Focusing on what remains or becomes a public name would be useful. stride_tricks indeed, but there is more. array_utils is new. What about index_tricks, I think we were considering keeping that and exposing some of the more niche or advanced indexing related utilities there?

The checklist of what to make private is useful too, but that's more a todo list. I'd suggest going into more detail on what will be officially public. And then on what the TBDs are or potential pain points. If some current .py file contains only objects that are in the main namespace already, then it's easy - that .py file does not need to be a public module. When that's not the case, we have to surface the changes.

@rgommers
Copy link
Member

I am not sure, but I suspect that normalize_axis_tuple being in stride_tricks was an accident - it has nothing to do with strides, and I'd expect it in another place (array_utils maybe?).

@mhvk
Copy link
Contributor

mhvk commented Aug 23, 2023

normalize_axis_tuple is imported from numpy.core.numeric, no worries about that (though there was a separate question of where it should be exposed - but not here for sure!)

@mtsokol
Copy link
Member Author

mtsokol commented Aug 24, 2023

The checklist of what to make private is useful too, but that's more a todo list. I'd suggest going into more detail on what will be officially public. And then on what the TBDs are or potential pain points. If some current .py file contains only objects that are in the main namespace already, then it's easy - that .py file does not need to be a public module. When that's not the case, we have to surface the changes.

@rgommers Sure! I updated the PR's description.
For each submodule, I defined what goes to the main namespace and what goes to the local one, as of now. It looks that the concept of public items available only in the local namespace applies only to stride_tricks, index_tricks and utils (and future array_utils).

All "main ns" public items are in the agreed "keep" list, so main namespaces shouldn't shrink. I think it's only local namespaces that we will discuss then.

This was referenced Aug 25, 2023
Merged
Merged
Merged
@rgommers
Copy link
Member

Thanks Mateusz, that's a clear list. There really is very little in "local ns", so basically we're hiding almost everything that's there now and have more or less a clean slate to add new niche functionality that doesn't qualify for inclusion in the main namespace. That'll be quite nice, and has been on our wish list for a long time.

Maybe add local ns items for numpy.lib itself? I think that there's a couple of items in it, like NumpyVersion and Arrayterator.

@mtsokol mtsokol mentioned this issue Aug 25, 2023
Merged
@ganesh-k13 ganesh-k13 mentioned this issue Aug 27, 2023
Merged
5 tasks
@rgommers rgommers mentioned this issue Aug 28, 2023
Closed
This was referenced Aug 28, 2023
Merged
Merged
Merged
Merged
Merged
Merged
Merged
@mtsokol mtsokol self-assigned this Aug 29, 2023
@rgommers rgommers mentioned this issue Aug 30, 2023
Closed
18 tasks
@ngoldbaum
Copy link
Member

It looks like there's still two minor usages of import * in numpy/lib/__init__.py that could be replaced but other than that this is done. Going to close this as I don't think those usages are bad, they're doing import * from modules with an __all__.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mtsokol mtsokol

Labels
Numpy 2.0 API Changes
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rgommers @mhvk @ngoldbaum @mtsokol