Probably also figure out a nice api that we can use for SearchCV as well

I think that having a single scoring param able to handle constrained metrics as well would be ideal. In addition to make an example for that, we can maybe implement a helper for the main uses cases, precision at recall and tpr at tnr, that returns a ready-to-use scorer.

Probably also figure out a nice api that we can use for SearchCV as well

Do you think it would make sense to search over constraint values ? My understanding was that such constraint is determined by an external factor that you have no flexibility over.

At least in fairness related context, you might have a constraint on a fairness metric, and among those you want the best accuracy.

Or if you take the classical example, you want the model which has the least AIC/BIC within the 1 std of the test error of the best model.

would be interested in contributing to this. my understanding is that a short example should be added to show the current api can be used (although in a bit of a tricky manner, using -np.inf) to optimize a metric according to some constraint. for where to add it, some potential spots i identified could be here in 3.3.1.1, inside the note, or making a new subsection in here in 3.4.1. perhaps most ambitiously it could be inserted in this existing example on cost-sensitive learning.

I think we should not complexify the example that you pointed out. However the following one might be a better place where we can show the pattern and the associated ROC curve to understand the constraint.

Then, we can document in the user guide and refer to this example.

added a draft PR for this example, still need to do a little more writing but the code and layout changes and overall shape is what I have in mind for reworking this example to include the constrained metric pattern. please let me know your thoughts