✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: b2144ee. Link to the linter CI: here}

I'm +1 on this:

• I haven't seen anybody use it (except maybe @amueller ?)
• If we deprecate and people start complaining, we can revert deprecation. So in a sense deprecation is a way to check usage here
• The deprecation message clearly should always point to an alternative, which I think is the case here.

The deprecation message clearly should always point to an alternative, which I think is the case here.

I think that we should do a bit more and specify how you get a similar behaviour by setting the right parameters.

In fa3d842, I added C to SGDClassifier and SGDRegressor to make it more accessible.
I could imagine to put the equivalent PA estimators into the SGD docstring - maybe once the deprecation is carried out.

Nice. We need to add more information on the docstring about C though. It's very short and not clear what it does when I read it. Otherwise LGTM.

@lorentzenchr getting close to the release, would you mind resolving issues here?

@lorentzenchr getting close to the release, would you mind resolving issues here?

I can resolve merge conflicts, but I won't do

We need to add more information on the docstring about C though.

The decorator deprecated interferes with the signature, e.g. in check_do_not_raise_errors_in_init_or_set_params:

    params = signature(Estimator).parameters

I need help how to avoid that (I won't fix deprecated) or rather to (temporarily) switch of those tests for PassiveAggressiveXX.

Note that 306a5fa should be made a PR of its own in case this PR is not merged.

Edit: I opened #30145.

In both our pytest and doc build, we can set the warning filter for this specific warning to be ignored. That should fix the CI

Where can I filter warnings for the doc build?

While the CI failure of Ubuntu_Jammy_Jellyfish pymin_conda_forge_openblas_ubuntu_2204 is clearer because it sets "SKLEARN_WARNINGS_AS_ERRORS": "1" (not 100% clear though which option supersedes which), the failure of ci/circleci: doc is not clear to me. I added the filterwarning but it still errors. It does not print where and I can't reproduce locally => I need help with this.

Maybe add your warning to

This thing is used to centralize the warnings that we can not do much about both for the doc build and the tests.

Over the time with this PR, I become some sort of expert for the Passive-Aggressive algos with two insights:

1. Interestingly, PA can be generalized for all loss functions $\ell$ with minimum zero not just Hinge and epsilon insensitive loss (was unclear before, see WIP: Adding Passive Aggressive learning rates #1259 (comment)).
2. It can be formulated as a normal SGD scheme ($\tau$ in the Online Passive-Aggressive Algorithms becomes $\eta$ in SGD). This way it becomes clear that it actually minimizes a loss (unclear before, see WIP: Adding Passive Aggressive learning rates #1259 (comment) and WIP: Adding Passive Aggressive learning rates #1259 (comment))

Let $\ell(x\cdot w, y)$ be the loss for a single sample vector $x$ with weights $w$ and observed class $y$ with $\min \ell(x\cdot w, y) = 0$.

The main trick is to uses a linear approximation of the loss as $\ell(x\cdot w, y) = \ell(x\cdot w_t, y) + (w - w_t)\cdot x \ell^\prime(x\cdot w_t, y)$ (derivative w.r.t. to its first argument), which seems appropriate for the purpose of SGD.
Note that $\frac{d}{dw}\ell(x\cdot w, y) = x \ell^\prime(x\cdot w_t, y)$.

PA-I

(Eq. 6) Update current weights $w_t$ as $w_{t+1} = \mathrm{argmin}_w \frac{1}{2}\lVert w - w_t\rVert^2 + C\xi$ such that $\ell(x\cdot w, y)\leq\xi$ and $0\leq\xi$. C is a free positive parameter controlling the step size.

Solution: $w_{t+1} = w_t - \eta x \ell^\prime(x\cdot w_t, y)$ with $\eta=\min(C, \frac{\ell(x\cdot w_t, y)}{\lVert x\rVert^2\ell^\prime(x\cdot w_t, y)^2})$.

For Hinge loss with $y=\pm 1$, $\ell^{\prime 2} = y^2=1$ one recovers $\eta=\min(C, \frac{\ell(x\cdot w_t, y)}{\lVert x\rVert^2})$ of the paper (there called $\tau$) and update $w_{t+1} = w_t + \eta x y$.

PA-II

(Eq. 7) Update current weights $w_t$ as $w_{t+1} = \mathrm{argmin}_w \frac{1}{2}\lVert w - w_t\rVert^2 + C\xi^2$ such that $\ell(x\cdot w, y)\leq\xi$.

Solution: $w_{t+1} = w_t - \eta x \ell^\prime(x\cdot w_t, y)$ with $\eta=\frac{\ell(x\cdot w_t, y)}{\lVert x\rVert^2\ell^\prime(x\cdot w_t, y)^2 + \frac{1}{2C}}$.

I might consider a follow-up PR which will generalize, but actually simplify the SGD code.

Maybe for another PR: We could think of avoiding to introduce the new parameter PA_C and repurpose an existing parameter, e.g., eta0.

If any of the participating possible reviewers could give a 2nd approval, that would be great. I really would like to have this over the finish line before possible merge conflicts (for example #31856).

When it's merged, I might open another PR to use eta0 instead of PA_C. Then we have time until the 1.8 release. But I only have time until end of August.

@adam2392 or @OmarManzoor could maybe have a look?

@OmarManzoor You spotted an error in the docstring of pa2. Should be corrected with
b2144ee. Could you (auto-) merge again?