sounds like a reasonable addition. please open the PR. I can't guarantee that it'll get accepted but it makes sense to me.

This feature is also implemented in #9405, where you can exclude coefficients from the L1 as well as from the L2 penalty term.

If possible could penalty factors also be added to LogisticRegression (which is also part of glmnet)? It would also be very useful in the same way as for ElasticNet except for classification problems where you need to include unpenalized covariates.

How do we want the API?

1. Allow to specify an array-like for the penalization strength? Same for l1_ratio?
2. Could we already permit to work with feature names? Then a dict could work {"featue_1": 0.5, "features_2": 2.5}

Among the estimators are:

• LogisticRegression(C=, l1_ratio=)
• Lasso(alpha=)
• ElasticNet(alpha=, l1_ration=)
• Ridge(alpha=)

What do we do with the CV variants? I would leave them untouched for the moment.

@scikit-learn/core-devs as info

can I get a review on my pull request that addresses this issue?

Ridge already accepts alpha as an array of (n_targets,). If GLMs support an array of (n_features,) the two APIs would be inconsistent.

Can we accept shape (n_targets,), (1, n_targets), (n_features, 1) or (n_features, n_targets)?

Personally, I find the multi-output story for penalties a bit unfortunate (maybe I'm blind or unaware of good use cases). Given that we can't change that (easily), what about introducing new parameters P1 and P2 as in https://glum.readthedocs.io/en/latest/glm.html (and as in #9405) for penalty matrix for l1 and penalty matrix for l2.

In particular the P2 would be nice to have as it generalizes the penalty to coef @ P2 @ coef. This can come in very handy, e.g. for penalizing differences of coefficients or just for different penalty strength per feature.

Further advantages:

• unified way to specify feature-wise penalties across all (linear) estimators: always the same parameter name
• clear distinction between l1 and l2
• no mixing/confusion with n_targets

What is n_targets? is that not the number of features?

It's the number of targets in y:

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge

X, y = make_regression(n_targets=2, random_state=0)

print(y.shape)
# (100, 2)

reg =  Ridge().fit(X, y)
print(reg.coef_.shape)
# (2, 100)

It's the number of targets in y:

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge

X, y = make_regression(n_targets=2, random_state=0)

print(y.shape)
# (100, 2)

reg =  Ridge().fit(X, y)
print(reg.coef_.shape)
# (2, 100)

ah ok thanks, haven't ever had the need to fit a regression w/ multiple targets before

But to me P1 cannot be a generic matrix as otherwise
the solvers will be a lot more complicated.

Correct, a 1d array for the diagonals of a P1 matrix suffices and keeps things solvable within our codebase.

As discussed in the dev meeting 28 March 2022, new parameter(s) seems fine. The questions are:

1. One or two parameters?
2. Naming

One vs two

Having one parameter like alpha_features would work for pretty much all linear models. This keeps L1 and L2 penalties in sync.
Having two individual ones, like P1 and P2 for ||P1 * w||_1 and ||w' @ diag(P2) @ w||, would allow more control. It also opens the opportunity for later to allow P2 to be a 2d array.

There are several use cases:

• I might not want to penalize some coefficients at all, in particular very strong main effects.
• I might consider to add L1 to all coefficients, but exclude main effects from L2 which acts stronger on large coefficients.
• With two different parameters for L1 and L2, we could later allow the L2 parameter to be a 2d array. With this one can construct:
  • L2 penalties of differences of coefficients like sum_{i>j} |w_i - w_j|, e.g. for splines (=> p-splines)
  • L2 penalties on differences of coefficients of neighbouring geo-locations, e.g. see section 6 of https://github.com/lorentzenchr/Tutorial_freMTPL2/blob/master/glm_freMTPL2_example.ipynb
  • Use more fantasy

Thanks for the effort on this so far. Could the new parameter(s) for flexible penalization of coefficients also be added to other models using the elastic net regularizer?

Here's another paper that could add some motivation for selective penalization from single-cell RNAseq bioinformatics: https://doi.org/10.1093/bioinformatics/btac227.
The authors present a method based on the glmnet R-package that models a penalized ordinal logistic regression to identify descriptive genes in ordinal settings. It fits thresholds and coefficients at the same time, but only penalizes the coefficients with an L1 regularizer. They explicitly do not use a multiple logistic regression in order to model ordinal sequences of categories (e.g. time series).

In an attempt to make this method available in python, I saw the use case for coefficient-specific penalization in SGDClassifier.

I did this PR as part of a school project so I'm unable to devote anytime to this anymore.