@simonamaggio told me on a private channel that this is still work in progress: new tests are needed before reviewing.

Also, it would be great to check the computational benefit of this approach on a small benchmark snippet to check that the caching mechanism works as expected.

Thinking about it the caching will probably not work because of the Parallel call: the multimetric scorer keeps the cached predictions as an attribute on the scorer which is mutated the first time it is called. Here this will happen in isolated processes. I think we need to somehow inverse the two loops but I am not sure how...

Edit: The above is probably wrong as discussed below.

@thomasjpfan @NicolasHug It could be nice to have your thoughts regarding the way to output the feature importances with multiple metrics. We went for a dict of Bunch but maybe you have some thoughts about it?

We went for a dict of Bunch but maybe you have some thoughts about it?

Sounds like the most sensible way, from a quick glance. What else did you have in mind?

We went for a dict of Bunch but maybe you have some thoughts about it?

Sounds like the most sensible way, from a quick glance. What else did you have in mind?

In cross_validate the output is a dict with keys including the name of the specific metric: ret['test_%s' % name] = test_scores_dict[name]. Here it could be similarly 'importance_'%name. But that function returns values, not Bunch objects.

Fair point. I would sacrifice a bit of consistency for the sake of usability and keep the current dict of Bunch, which seems much more practical. In cross_validate, getting all the statistics for a given metric is a pain as one has to manipulate strings. Same if you want the test score of all metrics.

I would agree with the dict of bunches. On the topic of Bunch, would Bunch of Bunch be better?

I would agree with the dict of bunches. On the topic of Bunch, would Bunch of Bunch be better?

Maybe I'm missing something, but using a Bunch of Bunch it seems tricky to get the keys for the external Bunch, from the strings of the metric names. For instance if I use scoring = ['precision', 'recall'], I'd need to access the internal Bunches with result.precision and result.recall.

I would agree with the dict of bunches. On the topic of Bunch, would Bunch of Bunch be better?

Not sure it brings that much benefits as the list of keys is not static, so attribute based look-ups is not necessarily that natural. But not a strong opinion.

Maybe I'm missing something, but using a Bunch of Bunch it seems tricky to get the keys for the external Bunch, from the strings of the metric names. For instance if I use scoring = ['precision', 'recall'], I'd need to access the internal Bunches with result.precision and result.recall.

You can also access those results using the usual dict API as Bunch is a subclass of dict.

All CIs are green. Merging. Thank you @simonamaggio

@glemaitre @simonamaggio we forgot to document the change in doc/whats_new/v1.0.rst. @simonamaggio would you mind opening a new PR for this?

Scratch that, we did. My ctrl-f did not work as intended...

Scratch that, we did. My ctrl-f did not work as intended...

Hehehe. This could have happen thought, it is quite common that I forgot it during the review :)

I understand why my ctrl f was not working anymore in firefox: I had the match "Whole Words" option enabled without realizing it...