That last case is still pretty ugly, and I'd be tempted to change it to

masked_array(
 data = [[--, 0],
         [0, 1]],
 mask = [[ True, False],
         [False, False]],
 fill_value = 999999)

That is, move the kwargs to the start of the line at the expense of not aligning them.

almost makes it valid python

If we replaced the -- with __, then it would be fully evaluable with eval(repr(arr), vars(np.ma)). It'd then be reasonable to have np.ma.__ an alias to np.ma.masked

I'm thinking about it :)

This seem like right time for any changes, but we have to balance it somehow to avoid making arbitrary changes that just annoy people.

Also on my mind is the fact that we might start a new maskedarray implementation. If some of these changes are too big to put here, we might put them in a new implementation.

In specifics: I like the __ idea a lot, but it's probably too different from the current output. It seems like a great idea for a reimplementation. Adding commas seems like a very desirable change, so much that I might almost be in favor of breacing doctests for them, not sure.

The newline changes might be OK.

Another idea: the mask = part of the repr seems redundant since masked values already have '--' in the data part, so maybe one day we could remove that entirely. But maybe it is there to avoid confusion involving nomask.

Re: A reimplementation of MaskedArray:

I think better missing-value support could be a big boost for numpy. It is quite poor at it compared to other languages like R. Wes McKinney even mentioned missing value support in his recent blog post justifying his desire to move pandas to use apache arrow instead of numpy (though I think pandas does its own thing with nans right now).

I am aware there are a bunch of different ways we could implement missing value support, but a well though-out MaskedArray implementation seems like an easy one, given we can work with the current MaskedArray code. There are ways in which I think a good MaskedArray implementation could be better than the alternatives such as the sentinel values used in R.

I think __array_ufunc__ is in now, so we could try using it. I just spent an hour going through the maskedarray code and removed a ton of code related to nomask, hardmask, sharemask, and removing functions that seem redundant. Lots of fun! But it's in heavy dissaray and not compiling now, the only code I have actually written is a little in in __new__, __array_finalize__, __getitem__. (Probably not worth looking at at this point, but I put the files here.. feel free to try from scratch too)

I like the __ idea a lot, but it's probably too different from the current output.

Note that this is already configurable with np.ma.masked_print_option.set_display('__')

justifying his desire to move pandas to use apache arrow

And he is one of the designers of apache arrow. My guess is that pandas will make the move at some point in the future as apache arrow was in part motivated by a desire to have something better adapted to big, really big, tabular data.

I'm thinking about what the role of NumPy will be in a future where pandas has moved on and multi-terabyte data is commonly worked with. I also recall @mwiebe's comment that NumPy is quite slow and inefficient. Some things can be improved, but to some extent we will be locked in by current functionality, striding and such. I do think we need better strings, dtypes, and missing value handling.

@charris, What comment was that? I don't recall. And what is the stride problem, again?

Numpy's speed and low abstraction level are definitely some of my main motivations for using it. I basically use it as a way to get near-C speeds, but with all of python's convenience. I've been vaguely keeping track of cases in my programs that I felt more numpy optimizations were possible, mainly cases we might avoid temporary arrays or add short-circuiting.... but I'm interested to hear more.

Also, numpy has always seemed to me like the lowlevel base for other projects like scipy, dask and numba, and I don't see it ever really going away. The thing that isn't totally clear to me is numpy's relation to projects like dynd, blaze, plures. I sort of thought they were attempts at numpy 2.0 which would solve all our legacy problems, and we would all transition to them, but they seem slow in coming. Or maybe the way to view them is as experiments, large parts of which we might paste in into numpy, or build numpy on them.

As for pandas, I think it is pretty natural for it to move to arrow. Pandas always seemed to me much more like a relational database interface for python. (I actually don't use pandas much since I don't work with very relational data).

What comment was that? I don't recall. And what is the stride problem, again?

Mark made the comment some years ago when Julian started making some of his speed improvements using vectorized instructions. The problem with strides is that they can break locality so that cache is not used efficiently. There is really no easy way around that as strides and subarrays are one of the most useful features of NumPy.

I'm thinking about sustainability. Technical obsolescence is certainly one way to lose a niche, OTOH, there are limits to what can be done with NumPy and still be NumPy. So I'm trying to clarify for myself exactly where NumPy fits in.

I think DyND was effectively an attempt at NumPy 2.0, but NumPy was too established at that point for DyND to really gain a foothold. Unfortunate timing, really. However, it has explored some good ideas that we should take a look at.

Ah, perhaps you are referring to his NEP for a Optimizing Iterator/UFunc Performance (link). I'll read it through.

By the way, I notice again that there is an NEP for a groupby function. We had a discussion about this a year ago on the mailing list, and the conclusion was "don't try it, we don't want to step on panda's toes". We should think about it again if pandas goes its own way.

Any decision on this?

Another comment on masked array. There was an attempt to make masks part of ndarray that ended getting shot down on account of the basic C structure getting two new items. I don't think that is considered a relevant criterion these days and we might want to revisit the idea.

As to this PR, a decision one way or the other would be helpful as I would like to branch 1.14 in the not too distant future.

Involved in the decision, thoughts on this comment would be appreciated too.

It's very tough to decide these things because the costs and benefits are hard to weigh.

In the "for" camp, I personally think the proposed modifications are nicer. I like the version which is pushed to the left. But should we go further? For instance, I might prefer not to print out the mask= at all, since that info is already visible through the location of the '--' in the data, and then we could get rid of the data=prefix and just print the data array.

In the "against" camp, the old output isn't that bad. Also, I think there is less motivation for this than for the ndarray printing changes we have made so far. For the ndarray changes we managed to make a "legacy" mode which preserved the old behavior, but we don't have that here. Judging from comments we got, some people do care about having a legacy mode. Also, the ndarray printing changes had a clearer goal to fix inconsistencies in array printing which were impeding other bug fixes and improvements. In contrast the changes here don't really "fix" anything, they are just prettier.

This is a change that, if we were working on a "numpy 2.0" branch, I would be all for. But after writing the comments above, I think the lack of a legacy mode might push me over to the "no" side. I might be convinced otherwise.

@eric-wieser what do you think?

Although I think it is an improvement, I'd also hesitate to put this in... Especially since I agree there is no real need to display the mask. But it does depend a bit on what we think the likelihood of a ma 2.0 is..

A propos ma 2.0, I just happened to think about it yesterday: it might be easiest to split it in two cases, one that is strictly a container class that propagates a mask with the same shape as whatever the data is (i.e., or the mask for any binary operation; propagate it for mean, etc.; here, the mask behaves like a nan), and a subclass specific to numerical data that has the logic of masking inf, nan etc. (and doing the equivalent of nanmean etc.). With __array_ufunc__ the first should be nearly trivial, and the second a modest addition.
There might be some case to be made for making the numerical one work with float only (so one can use things like nanmean instead of rewriting averaging routines).

That's fun - github is showing those in date order, not DAG order... The two MAINT commits are extracted to #9913, anyway.

For instance, I might prefer not to print out the mask= at all, since that info is already visible through the location of the '--' in the data, and then we could get rid of the data=prefix and just print the data array.

It's surprisingly hard to reconstruct a masked array from just the repr otherwise - np.ma.array([1, 2, np.ma.masked]) errors, for instance - so I think we need to keep the mask = until that's fixed

In the "against" camp, the old output isn't that bad.

These are pretty irritating:

>>> np.ma.array(np.eye(4))
masked_array(data =
 [[ 1.  0.  0.  0.]    #really?
 [ 0.  1.  0.  0.]
 [ 0.  0.  1.  0.]
 [ 0.  0.  0.  1.]],
             mask =
 False,
       fill_value = 1e+20)

>>> np.ma.array([datetime.now()]*2, dtype=object)
masked_array(data = [datetime.datetime(2017, 10, 23, 21, 25, 58, 273000)
 datetime.datetime(2017, 10, 23, 21, 25, 58, 273000)],  # why?
             mask = False,
       fill_value = ?)  # the heck is this?

when they could be

>>> np.ma.array(np.eye(4))
masked_array(data =
 [[ 1.  0.  0.  0.]
  [ 0.  1.  0.  0.]
  [ 0.  0.  1.  0.]
  [ 0.  0.  0.  1.]],
             mask =
 False,
       fill_value = 1e+20)

>>> np.ma.array([datetime.now()]*2, dtype=object)
masked_array(data = [datetime.datetime(2017, 10, 23, 21, 25, 58, 273000)
                     datetime.datetime(2017, 10, 23, 21, 25, 58, 273000)],
             mask = False,
       fill_value = '?')  # oh, it's a string

Hmm, I'm getting in the let's-do-it camp. But if we do it, I'd go all the way to putting data and mask on new lines (when needed), as you suggested (#9792 (comment))

(that was because I was too lazy to rebase)

go all the way to putting data and mask on new lines

I'm confused - the comment you link to is suggesting combining lines that are currently wrapped, not inserting more new lines

One thing that makes me more in favor is that it seems many of the people who care about doctests test continuously against numpy master. So if this PR is going to cause problems, we will know before the 1.14 release so will have a chance to reconsider. Also, assuming we don't make tons of MaskedArray changes before 1.14 release, this should be easy to revert.

It probably wouldn't be too hard to hijack the arrayprint legacy mode here to allow it to be disabled along with the other new behaviour - the diff is now not very big

@eric-wieser - I meant your suggestion to change multi-line reprs to

masked_array(
 data = [[--, 0],
         [0, 1]],
 mask = [[ True, False],
         [False, False]],
 fill_value = 999999)

(although I'd remove the spaces around = while you are at it...)

Where are we with this?

How do we feel about MaskedArray.__repr__ peeking at np.get_printoptions()['legacy'] in order to switch this behavior?

although I'd remove the spaces around = while you are at it...)

@mhvk: Should I do that for the "short" repr as well?

Yes, makes sense to look at get_printoptions -- supposedly that is how the user can know, so fine for us to use it internally too!

And, yes, I would remove them also in the short repr, it just looks more like an actual function call then, where spaces around equal signs are not PEP8y.

If this is made to depend on the legacy mode, I'll put it in. But be warned that I'm about to start pushing off the remaining PRs marked for 1.14 to 1.15.

Yeah, checking np.get_printoptions()['legacy'] seems totally fair.

Ok, overhauled with:

• Checking of `legacy == '1.13' that uses the old code path
• Addition of dtype=, since that was previously missing in important cases
• @mhvk's requested removal of spaces around equals (PEP8-inspired, I guess)
• The proposed new >1D format, which leaves data= at the start of the line.
• Short release note

I've updated the top comment with the new output, and added some more tests.

Besides the nd size=0 thing, looks good.

I think the current behavior is pretty good, and the 1d vs 2d difference works for me. The fact that this now line-wraps properly for long arrays is a big improvement! (I might even add a test for that).

One possible change: Use long mode for 1d arrays if they don't fit in a signle line.

One possible change: Use long mode for 1d arrays if they don't fit in a signle line.

This requires us to stringify the array twice, once for each indentation level, which will be twice as slow on large arrays. If you think this is acceptable, then I'll go for it

The fact that this now line-wraps properly for long arrays is a big improvement! (I might even add a test for that).

I think there already is at least one such test Good catch, that path is not tested - added now

One possible change: Use long mode for 1d arrays if they don't fit in a signle line.

This requires us to stringify the array twice, once for each indentation level, which will be twice as slow on large arrays. If you think this is acceptable, then I'll go for it

That's true. I'll leave it to your discretion.

There might be a way to avoid most of the cost with some simple length check. Eg, if there are more than max_line_width/2 elements use the long form, since each element comes with at least a comma and a space. That might be a bit too hacky though. (I mean, only do the double-stringify check after the length check tells you the short-form might be possible).

Rebased to use the cleaned-up typename calculations from #10032.

Have slightly improved the wrapping logic to allow any single-row array to appear on one line, but haven't gone as far as doing the double-stringification version.

The single-row change looks good, LGTM too.

I'll merge then, thanks Eric!