Hi @ArturoAmorQ, thanks for your review! I have made your suggested changes and resolved conflicts with the main branch. The test cases have passed, but I haven't tested for the error messages I added yet. Please let me know if I should make any further changes and if I need to modify or test for those error messages.

EDIT: Clearly I would need to add test cases for the error messages since Codecov did not pass. Will do soon.

Hi @glemaitre, thanks for your review! I've made your suggested changes. Please let me know if there are any further modifications you want me to make.

Thanks for the review @glemaitre @betatim! I have made the suggested changes, except that I have not added the new test yet. I will do that as well as adding this new feature in some example ASAP.

I have added this new feature into some of the examples @glemaitre. For instance,

As you can see, then chance level line is always at the bottom. I think we may need to fix both axes to [0, 1], as suggested in Issue #25929. If you agree, I will open another PR for these visual changes (for both PR and ROC curve), similar to what I've done in PR #25972, then apply them to these examples.

I think we may need to fix both axes to [0, 1]

Yes we will need the other improvement but in another PR.

Yes we will need the other improvement but in another PR.

Okay, will do after this PR is accepted (since I want to avoid too many merge conflicts).

Hi @glemaitre, thanks for your review! One thing is that Counter does not have total, and as suggested in the Counter docs, the common pattern for obtaining total counts is sum(c.values()), as currently implemented. Please let me know if this is okay.

True, this is Python 3.10. So let's revert then.

ping @betatim would you mind having a new look at this PR? This is good to be merged on my side.

Thanks for the additional exception. Merged.

Nice. Thanks @Charlie-XIAO

I am confused - what exactly does this line represent? The right-most point is the always-positive classifier as I understand it, which is indeed the baseline to beat in PR analysis - but it is not in any way random (and thus 'chance level' seems misleading in this context)? And how / why is the plotted horizontal extension of this point meaningful? Kull and Flach argue (see link above) that the comparison line to plot (analogous to the diagonal in the ROC curve) would be the F1 isometric curve associated with this classifier, which is hyperbolic and can also easily be computed:

  baserate = y_true.sum() / len(y_true)
  F1_baseline = 2 * (baserate * 1) / (baserate + 1)
  recall = np.arange(F1_baseline / (2 - F1_baseline), 1+1e-7, 0.05)  
  prec_baseline = F1_baseline * recall / (2 * recall - F1_baseline)
  plt.plot(recall, prec_baseline, 'k--', label="$F_1$ baseline")