Thanks for the bug report. Indeed, the data format should not affect the model. In your example, this issue is due to a combination of having a stochastic solver, a small sparse dataset, and a relatively large intercept.

Explanations:

• When solving a linear model with a stochastic solver, each weights update is proportional to the randomly drawn sample X_i. Thus, if the data is sparse, the weights are only updated on features j where X_ij != 0. On the contrary, the intercept is updated at each iteration, which can create large oscillations of the intercept over the solver iterations. Thus, a useful heuristic is to reduce the size of the intercept update (in stochastic solvers with sparse data), for example by a factor 100. This is set by the hard-coded parameter SPARSE_INTERCEPT_DECAY = 0.01.

• However, on some datasets, the model weights can converge in few iterations, and do not let the time to the intercept to properly converge. This risk is increased if the dataset is small, if the intercept is relatively large, or if the data is not very sparse.

Related issues:

• FIX switch to 'sparse_cg' solver in Ridge when X is sparse and fitting intercept #13995
• SGDRegressor gets poor fit with sparse matrix #11178
• [WIP] Common test for equivalence between sparse and dense matrices. #13246

Quick fix 1: Use the "lbfgs" solver on small datasets, add a warning when using "sag" or "saga" with sparse data and fit_intercept=True.

Quick fix 2: Use sklearn.linear_model._base.SPARSE_INTERCEPT_DECAY = 1. Warning: This quick fix is a hack. It uses private API that is not guaranteed to work in future versions, and it might decrease performances of other models.

Slow fix: To properly solve this issue, we could consider one of these options:

• A. Make SPARSE_INTERCEPT_DECAY a user-faced parameter.
• B. Change the heuristic for SPARSE_INTERCEPT_DECAY, or even remove it altogether.
• C. Change the stopping criterion to be more conservative about intercept changes.

Thanks Tom - this was very helpful. The quick fix 2 does remove differences in the models resulting from training on sparse v dense data.

@TomDLT Regarding slow fix B:

B. Change the heuristic for SPARSE_INTERCEPT_DECAY, or even remove it altogether.

Can the intercept decay be dependent on the sparsity of the data in X? For example, if 0.05 of the data is nonzero, then only about 0.05 of the coefficients get updated at each iteration, so we set the intercept decay to 0.05.

That would make perfect sense. It has been proposed before, and the answer was #612 (comment):

Have you used that before? Does it give better results? If so, we should definitely choose the density instead of the magic constant.

A quick thought: By using the density we assume that - more or less - the sparsity pattern of all features are equal. This might not be the case for bag-of-words features (Zipfs law).

On the other hand, I've been using the magic constant for about 3 years now; it gave reasonable results (compared to no damping of the intercept update). We could make intercept_decay a parameter and use 1.0 as dense default and 0.01 or density as the sparse default. Sometimes it makes sense to change the intercept decay - e.g. when you have both (a large number of) sparse and (some) dense features.

For reference, the intercept decay comes from Léon Bottou:

The learning rate for the bias is multiplied by 0.01 because this frequently improves the condition number.

@thomasjpfan , @TomDLT

Can I work on this issue?