How to interpret an approval in combination with a "+0"? 😄
My motivation is to close #19615 and I think this PR does it.

How to interpret an approval in combination with a "+0"? 😄
My motivation is to close #19615 and I think this PR does it.

Ok +1 then ;)

@agramfort any opinion on this and #19615?

tol can mean many different things depending on the stopping criteria employed (gradient norm, gain on the train objective of one update etc.) So the argument of consistency among estimators is not very strong me. Now this being said 1e-3 seems high. @lorentzenchr did you benchmark on a few datasets the impact of this in terms of cross-val score and fit time?

The impact is as follows:

• auto: see lbfgs, sag, cholesky, sparse_cg
• svd: tol has no impact
• cholesky: tol has no impact
• lsqr: tol is set as atol and btol of lsmr, which specifies conditions involving the residual and norms of matrix and coefficients.
• sparse_cg: relative or absolute residual smaller tol
• sag and saga: change of coef smaller than tol
• lbfgs: (projected) gradient=residual smaller tol

It may be worth to add that to the docstring:smirk:

Here a simple benchmark:

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge


X, y = make_regression(
    n_samples=100_000,
    n_features=100,
    n_informative=100,
    bias=10,
    random_state=42,
)

def max_abs_diff(a, b):
    return np.max(np.abs(a/b - 1))

%time svd = Ridge(solver="svd").fit(X, y)

Wall time: 692 ms

%time lsqr_3 = Ridge(solver="lsqr", tol=1e-3).fit(X, y)

Wall time: 125 ms

%time lsqr_4 = Ridge(solver="lsqr", tol=1e-4).fit(X, y)

Wall time: 114 ms

max_abs_diff(lsqr_3.coef_, svd.coef_), max_abs_diff(lsqr_4.coef_, svd.coef_)

(8.287694934638878e-05, 2.798488532462784e-06)

Same game for all solvers

I don't understand the results for sparse_cg.

thanks @lorentzenchr

a few remarks:

• what you call here accuracy is R2 and if so the number are super low suggesting that despite the n_samples >> n_features it does not learn much
• the fact that the R2 are quiet different suggests solvers do not fully converge with a tol=1-4. The story may be simpler if the problem is easier and R2 is closer to 1.
• the fact that you tests with n_samples >> n_features is necessary for SAG and SAGA to be descent solvers but it makes the problem very well posed with excellent conditioning. Picture could be different with n_features > n_samples.

wdyt?

With accuracy, I meant max_abs_diff, i.e. the maximum of relative differences of coefficients wrt the solution by svd. It shows that tol=1e-4 comes much closer to the correct values. Only sparse_cg seems odd.
These results are good evidence that decreasing default tol from 1e-3 to 1e-4 is beneficial for the found solutions (closer to correct values) with a comparatively small impact on runtime.

It may be worth to add that to the docstring😏

+1

Thanks for the experiments @lorentzenchr. I agree with your conclusions.

Good job @lorentzenchr

thanks @lorentzenchr