+1 for the plan except the "other ideas". I don't think that adding L-BFGS is necessary. And the QR based solver seems also an overkill.

for 2 you say lsqr for sparse case?

for 2 you say lsqr for sparse case?

I am not sure what you mean. I edited the description to remove a previous version of item 2 because it was wrong (my tol was too lax and that was leading me to wrong conclusions).

I'm also on board with the Proposed plan of actions, and, like @agramfort, don't think that adding lbfgs is necessary. A few further comments:

• tol=1e-3 is indeed a bit large. On top, it means different things for different solvers.
• "cholesky" solves the normal equations, i.e. computes X.T @ X + penalty, and therefore doubles the condition number of X which directly impacts the precision of solvers. The penalty should help, indeed.
• You propose a QR based solution. I could imagine, solving [X, lambda*Identidy].T @ coef = [y, 0].T in some way could be interesting, as it avoids doubling the condition number.
• lsmr is competing with lsqr. But in the past, I could not get a clear picture, which one to prefer in general.
  Both first use a Golub-Kahan bidiagonalization and then Givens rotations (leading to a QR decomposition) and solving triangular linear systems is more or less all that is done.
• lsqr should also work with sparse input and fit_intercept. If that were the case already, I could imagine to remove "sparse_cg".

Also in my benchmarks, sparse_cg seemed to perform reasonably well, even on dense array data.

@ogrisel Which benchmark are you referring to?

remove sparse_cg ? I would think it's the right thing to do for sparse X no?

Sorry, that suggestion of mine was typed too quickly and too late in the evening. I mixed up _solve_sparse_cg with cg based optimization routines. However, https://web.stanford.edu/group/SOL/software/lsqr/ states:

It [LSQR --Ed.] is algebraically equivalent to applying CG to the normal equation ( (A^T A + \lambda^2 I) x = A^T b, ) but has better numerical properties, especially if (A) is ill-conditioned.

On the corresponding LSMR page:

Special feature: Both (|r|) and (|A^T r|) decrease monotonically, where (r = b - Ax) is the current residual. For LSQR, only (|r|) is monotonic. LSQR is recommended for compatible systems (Ax=b), but on least-squares problems with loose stopping tolerances, LSMR may be able to terminate significantly sooner than LSQR.

remove sparse_cg ? I would think it's the right thing to do for sparse X no?

Interactive tests that I did by tweaking the end of the gist linked in the description.

Regarding adding or not LBFGS as a solver to Ridge, I just came across a use-case to model a classic mechanical (based on Newton's second law) problem where weights of the linear model are expected to be positive. Right now, we allow adding positivity constraints in LinearRegression, Lasso, and ElasticNet. Ridge is the only lonely linear model that does not currently support this use-case.

If we would like to add support for such an option LBFGS might be the solver that we want to use for solving this type of problem. It would make our linear models consistent in this regard. However, it is true that one can still use an ElasticNet and cancelling the l1 penalty to achieve such use-case but I am not sure if there is a performance drawback (maybe LBFGS would be faster than the CG used in ElasticNet?).

@glemaitre @agramfort I opened #19772 for positive in Ridge. As you'll see there, I did not propose LBFGS as special linear algebra based options are available.

I dared to check the 2nd point with #22910. Those are pretty tight tests.

What is left to do, is maybe decreasing (making tighter) default tol. From 1e-3 to at least 1e-4 (same as in most others like Lasso, LogisticRegression, ..). tol is irrelevant for the following solvers:

• cholesky
• svd

Let me reopen this issue because I believe that the default choice of parameters for the solver of the Ridge estimator could still be improved. In particular the following point of the description was not addressed:

maybe we could conduct a more systematic evaluation of various datasets with different shapes, conditioning and regularization to select a better default than "cholesky" for dense data? "cholesky" is reputed to not be numerically stable, although I did not observe problems in my gist, maybe because of regularization. lsqr seems to be very promising, both in terms of speed and accuracy of the solution (according to my check_min thingy).

Based on complementary experiments conducted in #22855 (comment) it seems that we should probably use lsqr either by default or as fallback for cholesky when the problem is singular, at least for some problem shapes. Also the fallback for cholesky should raise a warning telling the user to either increase the penalization or switch to lsqr directly to silence the warning.

There were also other interesting potential improvements proposed by @lorentzenchr in #19615 (comment).

Related issue when alpha=0:

• BUG unpenalized Ridge does not give minimum norm solution #22947

"cholesky" is reputed to not be numerically stable

This statement is for the unpenalized case and might apply for tiny alpha as well.

I think lsqr/lsmr could qualify as good default, but we should also do some experiments for n_features > n_samples.