Skip to content

RFC: remove support of pos_label inference with str in brier_scorer_label #18307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
glemaitre opened this issue Aug 31, 2020 · 3 comments
Open

RFC: remove support of pos_label inference with str in brier_scorer_label #18307

glemaitre opened this issue Aug 31, 2020 · 3 comments
Labels
module:metrics

Comments

@glemaitre
Copy link
Member

Currently, the following list of metrics will raise an error if str are provided as labels and that pos_label is not defined and needed to compute the metrics:

  • average_precision_score
  • f1_score
  • fbeta_score
  • jaccard_score
  • precision_recall_curve
  • precision_score
  • recall_score
  • roc_curve

We try to make sure that the error message is consistent in #18192

brier_score_loss should supposedly follow the behaviour than the other metrics. However, it seems that up-to-now, it does some inference when labels are string.

Thus, 2 questions can be raised:

  • Is it normal that brier_score_loss can do such inference or should it raise a similar error as other metrics?
  • If it should raise an error, shall we open consider as a bug fix or make a deprecation cycle?
@glemaitre glemaitre mentioned this issue Aug 31, 2020
Merged
@lucyleeow
Copy link
Member

lucyleeow commented Aug 31, 2020 •
edited
Loading

Related: #10010 - brier_score_loss is (now) also the only one using default pos_label=None (instead of pos_label=1). It also behaves differently for int label values than the metrics mentioned above:

    # if pos_label=None, when y_true is in {-1, 1} or {0, 1},
    # pos_label is set to 1 (consistent with precision_recall_curve/roc_curve),
    # otherwise pos_label is set to the greater label
    # (different from precision_recall_curve/roc_curve,
    # the purpose is to keep backward compatibility).

@ogrisel
Copy link
Member

ogrisel commented Sep 7, 2020

I believe we should go through a deprecation cycle (and update the docstring accordingly) to make it consistent with the others. I see no reason why Brier score would be special.

@glemaitre
Copy link
Member Author

I would like #18183 to get in before to make the proper PR.

@lucyleeow lucyleeow mentioned this issue Sep 15, 2020
Closed
@lucyleeow lucyleeow mentioned this issue Oct 24, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module:metrics
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ogrisel @cmarmo @glemaitre @lucyleeow