I would move the With random predictions sentence later. Otherwise it seem like it relates to the definition. I would mention here that there are interpolated variance and we're not doing that. Actually, I would rephrase the sentence above that this score corresponds to the area under the precision-recall curve. Because that is not entirely true / depends a lot on how you define that integral. Plotting the precision-recall curve, it's natural to do linear interpolation for visualization purposes (even though that might be misleading). But that's not how we calculate.

Also add references to the "relationship of PR curve and ROC curve" paper and the IR book and the pascal VOC paper / code (saying it's different). And pointing out that linear interpolation leads to realizable classifiers for ROC but not PR curve.

Maybe also add a reference to http://papers.nips.cc/paper/5867-precision-recall-gain-curves-pr-analysis-done-right.pdf because it also argues against linear interpolation.

I support these changes and will make the revisions.

If we remove the AUCPR statement here, I recommend that somewhere we state what the canonical AUCPR calculation is in scikit-learn. A user who wants to report AUCPR should be able to quickly find the appropriate function. @amueller do you suggest that the auc function on the precision recall operating points is the best option?

Because linear interpolation does not correspond to a realizable classifier?

My main gripe is the corner case where a classifier predicts only a single score, or scores that are close enough to be treated as equivalent. Computing the area with the trapezoid rule in that case gives area > 0.5 to a poor classifier. I've seen it cause problems for students trying to do automated hyperparameter searches based on AUCPR.

But whatever is going to be the reference AUCPR value should be recommended here. If that's not AP, I'll change it.

why did you remove the references? I think these references should go into the user guide and we should explain exactly the relation between different approaches.

I originally found it confusing to reference alternative implementations without explicitly stating they different from the AP implementation here. I'll add them to the user guide. I could also add them back here and present them as alternative interpolated approaches.

I agree. References in the implementation should be specific to the implementation. The user guide can give more commentary.

maybe emphasize that they lead to different results?

I don't understand. AUC is area under the ROC curve, not PR curve.

@amueller Per This is a general function, given points on a curve and the Davis and Goadrich paper, my understanding was that AUC can refer to trapezoidal area under any curve. In the Davis and Goadrich terminology we can have AUC-ROC or AUC-PR.

What change do you recommend?

never mind, I think your version is fine.

I'm updating the roc_auc_score description to say it computes ROC AUC (the acronym used in _binary_roc_auc_score) instead of the more general AUC.

Maybe explicitly say that this is not interpolated and that this is different from computing an interpolated area under the precision curve, which can be too optimistic.

maybe also add roc_curve? but also fine as-is.

Can you use the RST citation syntax and use references in the text, so that someone reading it knows what citation belongs to what statement? wiki and IR book should go to the definition. Then you can say "some areas such as computer vision use the interpolated area under the precision-recall curve instead [cite pascal VOC]. This can lead to an overly optimistic measure of classifier performance and has been criticized in the literature [Davis, Flach]. Right now, average_precision_score does not implement any interpolated variant".

@amueller Yes, I can use RST citation syntax. I haven't used it before, but the docs make it look like I only need to add tags like [Everingham2010].

I can also modify the text as suggested. However, my understanding is that the Pascal VOC paper and Davis / Flach papers discuss two different concepts. The VOC paper presents the 11-point interpolated average precision. The Davis / Flach papers do not discuss that but instead criticize linear interpolation of a PR curve.

VOC doesn't use 11 point in their code

@amueller Commit 0c0f4bf includes all of the changes you recommended during your last review. I added citation links in model_evaluation.rst but used different phrasing. My reading is that [Manning2008] and [Everingham2010] discuss interpolated AP whereas [Davis2006] and [Flach2015] criticize linear interpolation for AUC of precision-recall curves.

I'll confirm the citation links work once the docs finish building.