✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 414a8a3. Link to the linter CI: here}

The scipy solver indeed passes when setting cond = max(X.shape) * np.finfo(X.dtype).eps (new numpy default) or just cond = np.finfo(X.dtype).resolution, but fails when using cond = np.finfo(X.dtype).eps.

The scipy solver indeed passes when setting cond = max(X.shape) * np.finfo(X.dtype).eps (new numpy default) or just cond = np.finfo(X.dtype).resolution, but fails when using cond = np.finfo(X.dtype).eps.

We probably need to run a few variations of test_linear_regression_sample_weight_consistency with different values of n_samples and n_features to make sure that our choice for cond is robust.

Indeed it's quite difficult to find a good robust choice for cond. For instance, cond=resolution will often fail for n_samples, n_features = 100, 100, but then if you increase cond it can become stable again. Testing locally, the numpy default max(X.shape)*eps seems robust to different data shapes.

There is some explanation on choosing this cutoff ratio for singular values in:
https://numpy.org/doc/stable/reference/generated/numpy.linalg.matrix_rank.html

The test fails with the data with shape (100, 100) with considerable discrepancies:

E           Max absolute difference among violations: 0.00012815
E           Max relative difference among violations: 0.02112615

I am surprised that using fit_intercept=False seems to make the problem worse.

Could you try to re-run with (r)cond=np.finfo(A.dtype).eps / 2. * np.sqrt(m + n + 1.) from the expected round-off error quoted in the docstring of the numpy.linalg.matrix_rank function? Both with numpy and scipy?

Note that the cond parameter of scipy is documented as:

cond float, optional

Cutoff for ‘small’ singular values; used to determine effective rank of a. Singular values smaller than cond * largest_singular_value are considered zero.

https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lstsq.html

This is not the same as the rcond parameter in numpy:

rcond float, optional

Cut-off ratio for small singular values of a. For the purposes of rank determination, singular values are treated as zero if they are smaller than rcond times the largest singular value of a. The default uses the machine precision times max(M, N).

https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html

The rcond parameter in numpy is a threshold relative to the largest singular value, while the cond threshold in scipy is independent of the magnitude of the largest singular value.

EDIT: I misread the scipy docstring. Both definitions match even if the name of the parameter is different.

I have the feeling that only the numpy parametrization will make it possible to achieve what we are looking for.

The test fails with the data with shape (100, 100) with considerable discrepancies:

Actually, the same tests pass with dense numpy arrays. So the problem appears to be specific to the use of scipy sparse datastructures. I realize that for the sparse case, we use the "lsqr" solver that has no direct equivalent to the (r)cond parameter of lstsq because it does not rely on an explicit SVD internally.

So we should probably focus this particular PR to the fix for the dense case and open a dedicated issue+PR for the fix of the sparse case.

The added xfail tag (on csr array) is not ideal, because the test is actually passing for X_shape=(10,5) and for 98/100 seeds for X_shape=(10,20). It is always failing for X_shape=(100,100). But I guess it still gives a reminder to fix #30131.

Could you try to re-run with (r)cond=np.finfo(A.dtype).eps / 2. * np.sqrt(m + n + 1.) from the expected round-off error quoted in the docstring of the numpy.linalg.matrix_rank function? Both with numpy and scipy?

It fails with (r)cond=np.finfo(A.dtype).eps / 2. * np.sqrt(m + n + 1.) for both scipy and numpy on X_shape=(100,100).

@antoinebaker this PR is still marked as draft, but I have the feeling that it is now ready for review (actually, I just did). Could you confirm?

can I work?