DOC more precise calibration wording #28171
Conversation
A logistic regression will return well-calibrated predictions only if it is well specified. Some people were interpreting our text out of context and over-generalizing the sentence saying the a log-reg returns well-calibrated predictions. Here is a minor modification to make it more precise
There was a problem hiding this comment.
Just a tweak, otherwise LGTM :) Thanks @GaelVaroquaux, this certainly improves the wording!
There was a problem hiding this comment.
I agree with the suggested improvement.
Note that the level of regularization (and the presence of uninformative features in the low sample size regime) can also strongly impact the over-/under-confidence in predictions by
logistic regression models.
I plan to expand an example in the future to demonstrate this empirically but maybe in the short term we could just expand the paragraph as in the following suggestion.
But if you think that renders the paragraph too verbose, we can leave this for later. The current state of this PR is already a net improvement.
|
/cc @lorentzenchr |
doc/modules/calibration.rst
Outdated
| @@ -74,10 +74,12 @@ by showing the number of samples in each predicted probability bin. | |||
|
|
|||
| .. currentmodule:: sklearn.linear_model | |||
|
|
|||
| :class:`LogisticRegression` returns well calibrated predictions by default as it has a | |||
| :class:`LogisticRegression` is more likely to return well calibrated predictions by default as it has a | |||
There was a problem hiding this comment.
"more likely" calls for a comparison: more likely than what?
I would also like to change the term "by default" to something better. Maybe "by itself", "without re-calibration, or "if well specified"
Co-authored-by: Arturo Amor <86408019+ArturoAmorQ@users.noreply.github.com>
Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>
Merge suggested wordings from @lorentzenchr and @ogrisel
| @@ -74,10 +74,14 @@ by showing the number of samples in each predicted probability bin. | |||
|
|
|||
| .. currentmodule:: sklearn.linear_model | |||
|
|
|||
| :class:`LogisticRegression` returns well calibrated predictions by default as it has a | |||
| :class:`LogisticRegression` is more likely to return well calibrated predictions by itself as it has a | |||
There was a problem hiding this comment.
Maybe to address @lorentzenchr's comment:
| :class:`LogisticRegression` is more likely to return well calibrated predictions by itself as it has a | |
| :class:`LogisticRegression` is likely to return well calibrated predictions by itself as it has a |
There was a problem hiding this comment.
I'd rather not: some people are going to argue that it's not that likely in absolute
|
@GaelVaroquaux @lorentzenchr @ArturoAmorQ if you all agree with the suggestions in the comments above I can take care of applying this them all into this PR and rewrap the resulting paragraphs to get a clean diff. |
|
Superseded by #28231 |
|
@lorentzenchr not sure we wanted to close this one as it concerns the user guide, whereas #28231 targets the Comparison of Calibration of Classifiers example. |
|
I agree that this addresses a complementary aspect than #28231 and I think this needs to be merged. I must say that I got a bit tired by the back and forth of nitpick on wordings that are somewhat at odds with the fact that the original wording is overall not great and that, IMHO, this was an improvement. |
Exactly, my objective in opening #28231 was:
Still I think we should still fix the user guide to merge this PR possibly as is (as it's already a net improvement to mention well-specification which is a very important condition to get well-calibrated LR models. If you agree, I would be willing to put some further efforts to refine this section in a follow-up PR. |
|
If you agree, I would be willing to put some further efforts to refine this section in a follow-up PR.
Fine with me (of course :) )
|
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
|
So for closing too prematurely. But the net result is good. This PR moves forward. |
|
So shall we merge? My +1 still holds. |
| This leads to the so-called **balance property**, see [8]_ and | ||
| :ref:`Logistic_regression`. | ||
| In the unpenalized case, this leads to the so-called **balance property**, see [8]_ and :ref:`Logistic_regression`. | ||
| In the plot above, data is generated according to a linear mechanism, which is |
There was a problem hiding this comment.
@ogrisel Can you comment if this is true?
There was a problem hiding this comment.
It's probably dependent on the random seed but it's probably not too false, otherwise the calibration curve would be bad. I would rather keep the message simple enough in the user guide for now and refine the example(s) iteratively in follow-up PRs and update the user guide accordingly once this is done.
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
|
Thanks all. I will try to open a follow-up issue / PR soon. |
|
Thanks everyone!!
|
A logistic regression will return well-calibrated predictions only if it is well specified. Some people were interpreting our text out of context and over-generalizing the sentence saying the a log-reg returns well-calibrated predictions.
Here is a minor modification to make it more precise