I think the idea was more to use cross_val_score(GridSearchCV(...)).

Okay! There's a comment included at the bottom of the code that shows that the cross_val_score(GridSearchCV(...)) way. I thought it may be helpful to show the long way to show how to get the optimized hyperparameters on each inner iteration. Would it be preferred to remove that? I'm afraid it would be too short of an example; is anything else I can add in addition?

Thanks for the PR :)

Yes like Andy suggests conveying that doing nested cv has become simpler due to our new cv iterators could be more helpful.

I think using using LOLO-cv would be a good idea as you can show that we can pass labels for nested cv... Nested cv using KFold can be done with v0.17 itself.

I remember speaking with @ogrisel IRL at Paris Pydata. He suggested a good toy problem would be to predict the sepal/petal length/width using the other 3 features and use the iris classes as group labels. With this you can show that it is useful to use LOLO cross-validator as the measurements would depend on the iris class.

I feel that task will be confusing to people new to ml / sklearn.

Alright. In that case @mlliou112, you could additionally show how the scores without nested cv are not reliable and how nested cv reveals that inconsistency in scores for different parameter setting. That would be a nice way to highlight the importance of nested cv...

Great, sounds good. I'll work on it and push changes when I have them.

On a separate note, is there any reason you would suspect for the test failing on CircleCI? Also, is this something I should be worried about and how should I approach fixing it?

Alright, sorry for the delay. In this change, I've tried to illustrate here the slight optimistic bias of non-nested CV versus nested CV, especially when the splits are on a small dataset such as the iris one. I tried to narrow down how many parameter values are optimized over on GridSearchCV to cut down on running time.

Don't worry about the delay. This looks okay, though I'm not sure we should give the impression that those differences are significant. The text is also a bit verbose. The thing to emphasise is that taking the max over multiple parameter settings in grid search is liable to over-fit, often yielding an over-estimate of generalisation error.

I trimmed the text to make it more concise. Let me know what you think!

I'll take a look at this later, but it looks like your example is failing tests.

I don't want to make the text too long, but I think there should be a connection to doing a "train/test" split and doing a "train/validation/test" split.

Also, it might be important to point out what the result of nested cross-validation is. It doesn't yield a model or even a best parameter setting, so you don't get a model that you could use on new data.

It approximates the generalization error of the GridSearchCV estimator. I'm not sure there is a better way to say that. Can someone come up with a good use-case of this?

I was looking for a demo like this, so I like it! Then wanted to take it one step further and shuffled the labels expecting that nested_scores.mean() would be close to 1/3. Is my expectation wrong?

The original use-case that made me think about this is:

1. set aside a test set,
2. do *SearchCV on the rest
3. predict your leaderboard score using the test set, if higher than current submit to kaggle

Later on return to repeat steps 2. and 3.

However that setup is subtly different from the one in this example. So yeah.

@betatim Your expectation is correct. What numbers are you getting? I added a few lines of np.random.shuffle(y_iris) after loading the dataset, and the mean is reasonably around 1/3. nested_scores.mean() will have high variance since the dataset is relatively small and has only 3 labels.

@amueller I added two sentences with your suggestions. Let me know if they are appropriately placed!

Trying various seeds it fluctuates up and down. Should have tried that first. What I learnt from this is that sem(nested_scores) is not representative of the uncertainty of the mean, which is interesting.

I think this would be nice to have in 0.18. Could @jnothman mark this so?

This will likely miss the RC, @raghavrv, but we might be able to throw it in for final.

@mlliou112 you with us? :)

Yes! Following the 0.18 release has certainly been exciting. :) Is there anything else I have to do for this/or help in general?

You should refer to this example from the narrative docs on model selection.

@jnothman Thanks for the review, I took most your suggestions, w/ some minor alterations (see above). Let me know what you think.

otherwise LGTM

Aye, I think I messed up this branch/PR. There were new commits to cross_validation.rst and the KFold(...) method that affected this example so I tried to rebase the master commits.

How do I fix this? (And what was the better way to integrate those changes?) 🙏 Should I just make a new PR?

To fix the mess:

# go to your branch
git checkout your_branch
# always do a backup
git checkout -b your_branch_back_up
# squash your 9 commits into one
git rebase -i HEAD~9
# Now you have only one commit. Note its SHA (16 hexadecimal characters)
git log -n 1
# delete master
git branch -D master
# and download a fresh new version of master
git fetch upstream master:master
# go to master
git checkout master
# delete your branch (you have a backup)
git branch -D your_branch
# recreate your branch from fresh master
git checkout -b your_branch
# apply only your last commit with its SHA
git cherry-pick SHA
# force push
git push -f origin your_branch

In the future, when you need to rebase:

# go to master
git checkout master
# update master
git pull --rebase upstream master
# got to your branch (never work on master)
git checkout your_branch
# rebase your branch on master
git rebase master
# solve your conflicts and force push
git push -f origin your_branch

Highly recommend readings: http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html

@TomDLT Thanks very much! The development workflow article is much more helpful than anything that I found.

LGTM, thanks @mlliou112!

Merging. @amueller, please backport.

Thanks @mlliou112 :)

@TomDLT can you add the git stuff as a link to the contributing docs?

@amueller @mlliou112 I read the nested cross validation material (http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html) and it is really good! Do you know if I can get the best_params_ out of each fold of outer loop cross validation during the nested cross validation? I want to use nested cross validation to identify the best hyperparameters

@johnny5550822 The outer loop of the nested cross-validation is done only to evaluate the best models chosen by the inner loop. Refer this stackoverflow answer

@raghavrv ya, Hmm......so what should I do if I want to get the best hyperparameters?

To select the best hyper params, you simply do one cross-validation. To evaluate this 'selection', you do the outer cross-validation and infer if your selection can be trusted. Read the lower part of the above SO answer on what to look for in outer cv.

If from those inferences you understand that your selection can't be trusted, you tweak a different set of parameters / choose a different model / do more feature engineering.

Maybe @GaelVaroquaux @jnothman @amueller would be able to answer you in more detail. But unless I am mistaken, that is the crux of nested cv. Do the selection in inner cv, and use the outer cv to see if your selection can be trusted.

@raghavrv "To select the best hyper params, you simply do one cross-validation. To evaluate this 'selection', you do the outer cross-validation and infer if your selection can be trusted" Are you suggesting I use the inner-loop cross validation to identify the best hyperparameters? @GaelVaroquaux @jnothman @amueller

Are you suggesting I use the inner-loop cross validation to identify the best hyperparameters?

Yes...

@raghavrv but how can I do it in scikit-learn. In the nested cv example, seem like GridSearchCV is wrapped by cross_val_score. I don't know how to obtain the hyperparameters.

clf = GridSearchCV(estimator=svr, param_grid=p_grid, cv=inner_cv)
nested_score = cross_val_score(clf,scoring='roc_auc',X=features, y=labels, cv=outer_cv)
nested_scores[i] = nested_score.mean()

Just do clf.fit(features, labels).best_params_

(But please do wait for replies from others to gain more clarity...)

@jnothman To be clear, that is to say nested cross validation is not really the way to do hyperparameter selection (instead, we use repeated simple cross-validation). Rather, nested cross validation is to not only provide an estimated performance of the model, but also tell you roughly about "what is the simple cross-validation performance on hyperparameter selection?" Thus, the nested cross validation will give something like xx+-x%. Am I right?

Also, when we construct the nested CV, we pass the clf into cross_val_score (see below), there is no repeated trial inside the inner loop of CV right (i.e., just one time CV for each possible combination of parameter inside the GridSearchCV)? I

clf = GridSearchCV(estimator=svr, param_grid=p_grid, cv=inner_cv)
nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)

@johnny5550822 I think what you're saying is correct. I'd caution thinking about nested and simple as different methods of cross validation. Nested is really just the simple CV done twice for two things at once, estimating hyperparameters and evaluating parameter selection strategy. (I'll reemphasize that the "optimal parameters" given by the inner CV may not be the same for all outer CV loops.)

Yes, the 3-fold CV is just done once per combination of parameters, and gives out the average performance of those 3 folds.

@mlliou112 Great. I want to make sure the inner-loop is just done once because I originally thought it is done more than just one time. This is because the suggested nested CV is a repeated nested CV, i.e. the inner loop is repeated several times before identifying the best score. (https://jcheminf.springeropen.com/articles/10.1186/1758-2946-6-10) Maybe this can be part of the future work and allow user to use repeated or not.

@mlliou112

(I'll reemphasize that the "optimal parameters" given by the inner CV may not be the same for all outer CV loops.)

You are absolutely right. And that is exactly the problem. You cannot get the best parameters of a model, while using nested crossvalidation since the models could be different in each outer loop. A good way to visualize this is with this image I found https://mlr-org.github.io/mlr-tutorial/release/html/img/nested_resampling.png

Noone guaranties that the split of the inner CV will be the same in every outer CV loop, thus the hyper parameters and the model itself.

So I think there are two solutions:

1. you just do a GridSearchCV with a crossvalidation and you are done. You are left with the best model, best parameters, every developments set etc.
2. you do a nested cross validation and just get the mean scores.

Have I understood it correctly ???