Not sure this belongs to the highlights. This is already explained in the examples. And I mean all examples we show in the highlights are always not meaningful toy examples just for the purpose on simple demonstration.

I think optimizing for custom is the single most important use of this new tool. However the text and the code snippet do not make this point strong enough. I think we need to explain a minimum what a custom metric is and why it can often rely on side metadata to give motivation to the reader to follow the link and read until the section that explains cost-sensitive learning with side metadata.

I agree that this will be too complex to put in the code snippet of the release highlight but I thought this paragraph could help as a middle ground.

are always not meaningful toy examples just for the purpose on simple demonstration.

Doing things sub-optimally in the past should not be a reason to no improve for the future ;)

If the demonstration is not realistic enough, it's hard to assess the value of the new feature and justify spending time and effort to learn more in the doc / examples.

I think we need to explain a minimum what a custom metric is and why it can often rely on side metadata to give motivation to the reader to follow the link and read until the section that explains cost-sensitive learning with side metadata.

Alright

Doing things sub-optimally in the past should not be a reason to no improve for the future

I'm not saying it's suboptimal, to me it's good to make extra simple snippets :)

If the demonstration is not realistic enough, it's hard to assess the value of the new feature and justify spending time and effort to learn more in the doc / examples.

I'm not sure about that. I have the opposite impression. I'd be interested in seing statistics. To me, it drives more attention if it just shows that you can now do something that you couldn't do before in a very simple way. Then if someone wants to see how far he can go, he will dig the docs deeper.

Could we please use a meaningful custom metric then? And avoid all the balanced accuracy stuff?

Or simply use accuracy. Here for the threshold, it‘s fine as long as it’s not too extreme.

I'm okay with that

roc_auc_score should be invariant to a change in threshold.

Yes but I do not see how it's related to this discussion. Using a threshold-invariant metric is pretty useless to tune the decision threshold. Arguably we should even raise a warning in that in that case.

I would not mind to use a very short custom metric like.

Alright. I will update this PR accordingly then.

Yes but I do not see how it's related to this discussion. Using a threshold-invariant metric is pretty useless to tune the decision threshold. Arguably we should even raise a warning in that in that case.

you can ignore, I made a suggestion but quickly realised it was non-sense so I edited my comment 😄

I pushed 09bd60f. Hopefully it will help getting our message through while staying concise enough.

I changed the custom function to actually introduce a tradeoff between the two classes otherwise one would get a dummy classifier without realizing it.