• Can you make it a separate paragraph -- maybe starting *Compatibility guarantee:* or so
• And elaborate a bit to make it explicit that the guarantee is that for a fixed seed and fixed series of calls to RandomState methods it always returns the same results regardless of platform or numpy version, EXCEPT there might be slightly different rounding

This looks good to me.

There was some discussion on #5851 on whether this is the way to go, e.g. we now that dirichlet generates garbage results for certain parameters. Should we really make it part of the contract that we will not fix broken implementations if that requires changing the resulting state of the RNG?

I guess the rule has been, similar to the rules for deprecations,
transitioning from not-working-at-all -> doing-something-useful is
generally allowed. (Trivial case: adding a new method to the RNG object, so
before r.weirddist() was an error, but now it returns some random numbers
-- this is obviously legal even though technically it does change the RNG
output in some extreme rules-lawyer-ish sense.) We could document that here
if we want... the dirichlet case is a bit of an edge case, but IIRC does
kinda fall under this b/c it returns NaN in the bad cases?
.
Fortunately we have not yet encountered a case where a method returns
outright wrong results -- I'm not sure what we should if we ever do!

On Thu, Aug 20, 2015 at 12:28 AM, Jaime notifications@github.com wrote:

There was some discussion on #5851
#5851 on whether this is the way
to go, e.g. we now that dirichlet generates garbage results for certain
parameters. Should we really make it part of the contract that we will not
fix broken implementations if that requires changing the resulting state of
the RNG?

—
Reply to this email directly or view it on GitHub
#6180 (comment).

Nathaniel J. Smith -- http://vorpus.org

We've also fixed outright wrong results in the distributions before, and that has been our (my) stated policy.

Oh? All the more important that we document it then :-)

One possible policy going forward:

• We will fix outright wrong results,
• We won't bother keeping around and maintaining "compatability code" to let people reproduce the old wrong results with new versions of numpy,
• But, since people may nonetheless need to reproduce the old results (e.g., for reverse-engineering a published result), we will add a permanent note to the docstring noting the change and saying which versions were affected -- so similar to a .. versionadded::, except it tells you which versions were suspect.

does that sound good?

...and with the whole how-we-make-decisions discussion in mind, I guess it would be a good idea if whatever complete policy we come up with were sent to the list. I don't imagine anyone will object, esp. since it's mostly codifying how we've always done things, but it's good to do these things publicly.

This used to be documented on the Trac wiki back when we used that old thing, and I do restate our (my) policy whenever it comes up.

I like documenting when such necessary changes do happen. That's a good idea.

In addition, to be more specific about common cases:

• We allow widening the parameter domain such that values that used to be disallowed or trivially produced NaNs will do the right thing, provided that the results from previously-valid parameters are unchanged.
• We don't make otherwise correctness-neutral improvements if they change the results. E.g. standard_normal() will never use a ziggurat method :-)

Squashed and updated in #6405, closing this. Thanks @trankmichael .