@jnothman thnx a lot for the quick response. The visualisations I mentioned on the previous comment show just which point of the roc curve was chosen (for validating that the chosen point is the one closest to the ideal corner), so not a lot to see there. I plan on making a more extensive example and adding it here in the following days.

For your questions:

1. the labels can be basically arbitrary python objects, but you can assume there's exactly two of them. Check other classification code to see how mapping to 0 and 1 is handled.

2. it should behave consistent with other metrics, which assume 1 is positive if the labels are [0, 1] or [-1, +1] and otherwise require pos_label to be set.

3. That's what I would have done. That's somewhat different from what CallibratedClassifierCV does. I'm not sure if there's a standard behavior in the literature. I guess there is no really great way to accumulate the different thresholded classifiers. You could vote, but that sounds like it would lose information? But on the other hand, combining all the data together could completely change the meaning of the thresholds.... hum... In particular if you use a grouped CV I could see that going completely sideways.
   Can you do an example where different groups have different semantics for thresholds and see if your strategy breaks?

tests are failing

as you said in the other thread, we need a user guide and examples. Those often show whether the interface is good ;)

what's the state of this? I thought this was going to be included in 0.22

@marctorrellas I am sorry, I haven't caught up with the discussion as I didn't have the capacity to interact with this the past weeks. I may be able to find some time soon :|

I will take over the PR to address the remaining comments.

@glemaitre :) I haven't been able to dedicate any time in this for the past months due to job searching. Happy to see this completed!

Don't worry. I think that this an interesting feature and I want to give it a push.

Some thoughts that I had over time

• the interface wrt to strategy is a bit excessive. It should probably give only the option for tuning the fbeta.
• averaging the decision thresholds across folds, it is not super clear why this should work. Instead, we should probably select the decision threshold that works the best across folds?

the interface wrt to strategy is a bit excessive. It should probably give only the option for tuning the fbeta.

I would like to see if it could be useful in the case of class imbalance. Actually, we were interested of having a manual option with @ogrisel.

what will the manual option do? Allow the user to specify the exact fpr and trp they'd like?

sounds relevant, and more practical than the current interface - which in fact has a similar rationale behind it. I agree; I believe the default should be the fbeta.

• fold

I don't think that averaging thresholds is correct. For every training data there's an optimal threshold that will give best results on unseen data, and they can be quite different.

I think you're mixing two things:

• choosing the best threshold
• estimate performance for a model where we have selected the best threshold, i.e. evaluate "the strategy of train with a train set and selecting the threshold with a dev set using criteria X"

For the former I would recommend just train-dev split. Train on train, select threshold with dev

For the latter I would recommend nested CV, with the inner loop selecting the threshold, and the outer loop to evaluate. However, this won't give you which the model (the threshold) to use

I believe the default should be the fbeta.

I am not so sure. balanced accuracy could also be a good default but it would probably be more computationally intensive than f1 score. I don't have a strong opinion.

The classifier gives the option to either receive a pre-trained base estimator or train one and do the calibration using cross validation loops.

I am not so sure that cross-validation is really necessary here. As any of you tried or read references that would just fit the base estimator and tune the threshold on the same entire training set?

Balanced accuracy is usually not good for imbalanced datasets. I would give the option of f1 and balanced accuracy. They are not difficult / expensive to compute either

@ogrisel tuning on the training set is prone to overfitting, at least you need a dev set

@ogrisel tuning on the training set is prone to overfitting, at least you need a dev set

I was wondering if that would really be the case in practice, but maybe yes you are right: the roc curve could be very different on the training set than on a validation set for overfitting models and therefore using this for cut-off selection would result in a poor choice of the cut-off .

I don't understand why cutoff averaging across CV folds: if the cutoff selected on each CV split is not stable enough, then the cut-off Classifier procedure would be pointless anyway. Fixing an arbitrary internal train-validation split would not solve the unstable cut-off problem, it would just hide it.

(I'm curious, Mark, why balanced accuracy is not good for imbalanced datasets.)
…

My name is Marc :)

My statement was not exactly what I wanted to say. Balanced accuracy is better than naive accuracy indeed. I'd rather choose f1, as I see it more relevant for imbalanced datasets, and in the case of this PR let the user choose, with default for bacc maybe...

Example:
TP, FP 5 50
FN, TN 10 10000

balanced accuracy = 66 %
f1 = 14 %

I think in this case f1 represents better how bad this model is, but I'm sure other scenarios can show balanced accuracy as a good metric. Please let me know if there are any errors in the computations

@ogrisel tuning on the training set is prone to overfitting, at least you need a dev set

I was wondering if that would really be the case in practice, but maybe yes you are right: the roc curve could be very different on the training set than on a validation set for overfitting models and therefore using this for cut-off selection would result in a poor choice of the cut-off .

I don't understand why cutoff averaging across CV folds: if the cutoff selected on each CV split is not stable enough, then the cut-off Classifier procedure would be pointless anyway. Fixing an arbitrary internal train-validation split would not solve the unstable cut-off problem, it would just hide it.

You can see the decision threshold as a hyperparameter. Would you choose your HP on the training set?

I don't agree with the last bit. If the cut-off is stable, then either procedure is good. If it's unstable, then you need to select the one that is good for the training portion you have used.

Note: we assume it's unstable w.r.t. the training data you use, not the validation set to tune it

Thanks for the intuition, I think I get your point. I would still like to see simulations for various models (e.g. for overfitting and underfitting models of various kinds) to evaluate the impact of this strategy and in particular highlight catastrophic cases where CV-based cut-off averaging is detrimental. (if someone would like to volunteer to do a study in a notebook for instance).

In any case CV (+ refit on the full training set) is always significantly more computationally intensive than doing a single train-validation split (without refit on the full training set). So it's also a good reason not to do full-blown CV by default it never improves upon single split tuning.

Example:
TP, FP 5 50
FN, TN 10 10000
balanced accuracy = 66 %
f1 = 14 %

I don't see the issue with balanced accuracy. Basically 50% means randomness. So you are 16 % above randomness which is still not great as the 14 % of the f1 score is telling you.

why is f1 preferred over fbeta? Since f1 assumes that precision and recall are equally important for the user and fbeta with default beta = 1 is the same as f1, I'd think that fbeta makes more sense.

Example:
TP, FP 5 50
FN, TN 10 10000
balanced accuracy = 66 %
f1 = 14 %

I don't see the issue with balanced accuracy. Basically 50% means randomness. So you are 16 % above randomness which is still not great as the 14 % of the f1 score is telling you.

What I meant is that in the example above I think 14% reflects a lot better how bad the model is as compared to 66%, but of course those numbers explain better or worse depending on whether the reader understands the baselines and metrics. (Almost) any metric is good as long as you can interpret it correctly

About F1 vs Fbeta, I was just trying to reduce scope for the PR. I'm not against Fbeta

I'm closing this issue as superseded by #16525, following our triaging rules. Thanks @PGryllos for your work! Feel free to comment in #16525 if you are interested in a follow-up.