Depending on the background, many users expect logistic regression to not have a penalty by default.
We now have the option to turn off the penalty (#12860), but I think we should discuss this in more detail in the user guide.
Basically we should say something about it being regularized by default, potentially why that is (numerically easier and more stable is one reason), and what solvers allow turning off the regularization, and that that basically means setting C to a very large value.

Some discussion in #6738