This is only going to be simple usage of preprocessor and then talk about more details in the preprocessor section, right?

Yes, something like: input data: a 2D array-like X (or 1D array-like of indices, see preprocessor section)

So the goal is to do the following in order:

1. grid search for each metric learning algorithm
2. grid search for a classifier (same classification algorithm)
3. and then compare the results using classification metrics?

Right? Or are you only comparing the visualization after metric transformation (if so, maybe this could be combined with "Data visualisation" section)?

I was thinking only grid search for Weakly Supervised Algorithms like MMC, the score being the accuracy score on predicting if a pair is positive/negative, since this case is the most unusual for users used to classic train/test sets of points and not pairs. But indeed we could also make a grid search on pipelines (SupervisedMetricLearner, KNN) for instance, like a regular classifier, with a classification accuracy score.
And I don't know if you were refering to it by 1., but maybe for Supervised Learners we could also do a GridSearch with the scoring of Weakly Supervised Algorithms (we would split the points X traditionnally, but on the test set we would sample some pairs and then evaluate the accuracy of score_pairs) I don't know if this kind of scoring could is a traditional option for Supervised Metric Learners ? @bellet any opinion on this ?

And I agree that it would be interesting to see the impact of hyperparameters tuning on data visualization: maybe this could make an example in the data visualization section indeed

It might also be good to show a sklearn Pipeline

Yes I agree, since the pipeline (Metric Learner, KNN) for instance, represents a major use case of metric learning for classification

This example could be simplified a bit:

def find_images(file_paths):
   # each file contains a small image to use as an input datapoint
   return np.row_stack([imread(f).ravel() for f in file_paths])

nca = NCA(preprocessor=find_images)
nca.fit(['img01.png', 'img00.png', 'img02.png'], [1, 0, 1])

Yes, and your snippet with the imread etc is actually we would really do in a real use case

Done

You can re-use find_images from the other example above here.

Done

Maybe be explicit here that the X array is not used at all in this case.

I agree, I replaced it with a "not implemented yet" preprocessor, and wrote that the code would work anyways

...a gallery of example metric-learn use cases.

Thanks, done

This looks more suitable for the wiki, or maybe a master issue for improving docs.

yes, maybe you can put this in a wiki

Thanks, yes I forgot it here. That's right, I'll open a page on the wiki for that

raise instead of return

Right, thanks

maybe just add a sentence or two to briefly describe what the code snippet does (compute cross validation score of NCA on iris dataset)

Agreed, I've added a quick description and also modified the example since it didn't work in fact (for now we don't have a scoring for cross-validation on supervised metric learners), so I updated the example with a pipeline nca + knn

emphasize the scikit-learn compatible aspect

You're right, done

missing a few other generic methods, like score_pairs etc?

Indeed, thanks. I've added score_pairs and replaced transformer() by the newtransformer_from_metric(metric). I didn't put _prepare_inputs since it is private, and neither check_preprocessor because it is public for now but probably should rather be private ?

the fit method of the estimator?

I agree it may confuse users to say "the estimator will". I'll also say "predict etc", since preprocessors can also be used for prediction/scoring etc

done

maybe briefly explain default behavior (when preprocessor=None)

I agree, done

this is not the right reference (this is for ITML)

Thanks, didn't see that
Done

link to section does not seem to work (at least not on the version I compiled locally)

Thanks, that's because I forgot to put the right formatting for referencing sections :p
Done

This is Davis et al.

Indeed, thanks, done

the numbering of the section for each algorithm has an issue (e.g., shows 3.4.1. 1. ITML)

Thanks, done

it would be nice to have these references to link to the docstring page

Thanks, indeed I used the wrong way to reference them
Done

I suggest two changes here:

• "optimal" is not great (wrt what is it optimal?). I would instead say "to automatically construct task-specific distance metrics from (weakly) supervised data".
• the new quick start example below makes me realize that we should clearly mention the relation between distance metric learning and embedding/representation learning. We could add a sentence like:
  "The learned distance metric often corresponds to a Euclidean distance in a new embedding space, hence distance metric learning can be seen as a form of representation learning."

• "optimal" is not great (wrt what is it optimal?). I would instead say "to automatically construct task-specific distance metrics from (weakly) supervised data".

I agree, done

• the new quick start example below makes me realize that we should clearly mention the relation between distance metric learning and embedding/representation learning. We could add a sentence like:
  "The learned distance metric often corresponds to a Euclidean distance in a new embedding space, hence distance metric learning can be seen as a form of representation learning."

I agree, for now I just added this sentence, but indeed we should emphasize this more, maybe in the future with some section in the docs, and also maybe this will get clearer with examples of metric learners used as transformers (examples of dimensionality reduction for instance)

Maybe clarify the inputs to score_pairs, and "similarity of pairs of points" --> "distance between pairs of points"?

That's right, done