Nested cross-validation: why the proposed example does not have an hold-out set? #31510

SamGG · 2025-06-09T18:51:40Z

SamGG
Jun 9, 2025

I am learning Machine Learning and exploring nested cross-validation.

I don't understand the example given in scikit-learn. The model seems to learn from the whole dataset and the evaluation is not performed on a hold-out set.
scikit documentation
scikit implementation

# Loop for each trial
for i in range(NUM_TRIALS):
    # Choose cross-validation techniques for the inner and outer loops,
    # independently of the dataset.
    inner_cv = KFold(n_splits=4, shuffle=True, random_state=i)
    outer_cv = KFold(n_splits=4, shuffle=True, random_state=i)

    # Nested CV with parameter optimization
    clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
    nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)
    nested_scores[i] = nested_score.mean()

From what I read in Applied Predictive Modeling from Kuhn & Johnson, the model resulting from the inner loop should be evaluated on the hold-out set of the outer loop and the following post adheres to this point machinelearningmastery blog

As I am far from a Python expert, could you tell me the advantages, drawbacks and purposes of both of these implementations?

I read #21621 but I am not sure if it really answers my question. If it does, let me know and I will try to carefully understand it.

Mariihmp · 2025-08-07T10:43:54Z

Mariihmp
Aug 7, 2025

This Sckit-Learn way:
clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv) nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)

Is the same as this for loop way:

    #outer split
    X_outer_train, X_outer_test = X_iris[train_idx], X_iris[test_idx]
    y_outer_train, y_outer_test = y_iris[train_idx], y_iris[test_idx]
    
    #inner cv on outer_train set
    clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
    clf.fit(X_outer_train, y_outer_train)
    
    print({clf.best_params_})
    
    #evaluation on the hold-out set
    score = clf.score(X_outer_test, y_outer_test)
    scores.append(score)

After nested CV found scores around 95%-above means your GridSearchCV works well

0 replies

SamGG · 2025-08-10T19:11:25Z

SamGG
Aug 10, 2025
Author

Thanks for your feedback. Sorry for my slow understanding.
I don't understand where train_idx and test_idx came out in your example.
In the example, my point is that inner_cv returns indices that are present in outer_cv (indices are identical in the example due to the random_state initialisation). So, the score is evaluated on data that was seen during the GridSeachCV (even if random_state is really random). To me, this is not nested. Am I wrong? Could you elaborate an example to clearly express your point?

from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV, KFold, cross_val_score
from sklearn.svm import SVC

# Number of random trials
NUM_TRIALS = 1

# Load the dataset
iris = load_iris()
X_iris = iris.data
y_iris = iris.target

# Set up possible values of parameters to optimize over
p_grid = {"C": [1, 10, 100], "gamma": [0.01, 0.1]}

# We will use a Support Vector Classifier with "rbf" kernel
svm = SVC(kernel="rbf")

for i in range(NUM_TRIALS):
    # Choose cross-validation techniques for the inner and outer loops,
    # independently of the dataset.
    # E.g "GroupKFold", "LeaveOneOut", "LeaveOneGroupOut", etc.
    inner_cv = KFold(n_splits=4, shuffle=True, random_state=i)
    outer_cv = KFold(n_splits=4, shuffle=True, random_state=i)
    for j, (train_index, test_index) in enumerate(inner_cv.split(X_iris)):
      print(f"Inner Fold {j}:")
      print(f"  Train: index={train_index}")
      print(f"  Test:  index={test_index}")
    for j, (train_index, test_index) in enumerate(outer_cv.split(X_iris)):
      print(f"Outer Fold {j}:")
      print(f"  Train: index={train_index}")
      print(f"  Test:  index={test_index}")

Inner Fold 0:
  Train: index=[  0   1   2   3   4   5   6   9  10  11  12  13  14  15  17  19  20  21
  23  25  28  29  30  31  32  34  35  36  38  39  41  42  43  46  47  48
  49  50  52  53  55  56  57  58  60  61  64  65  67  68  69  70  72  74
  75  77  79  80  81  82  85  87  88  89  91  92  94  95  96  98  99 101
 102 103 104 105 106 108 109 110 111 112 113 115 116 117 118 119 120 122
 123 124 125 128 129 130 131 133 135 136 138 139 140 141 142 143 144 145
 146 147 148 149]
  Test:  index=[  7   8  16  18  22  24  26  27  33  37  40  44  45  51  54  59  62  63
  66  71  73  76  78  83  84  86  90  93  97 100 107 114 121 126 127 132
 134 137]
Inner Fold 1:
  Train: index=[  0   1   4   5   6   7   8   9  11  12  14  16  17  18  19  21  22  23
  24  25  26  27  28  29  31  32  33  34  35  36  37  38  39  40  41  42
  44  45  46  47  49  51  53  54  55  57  58  59  62  63  65  66  67  68
  70  71  72  73  74  75  76  77  78  79  81  82  83  84  86  87  88  90
  93  96  97  98  99 100 102 103 104 105 107 109 110 113 114 115 117 118
 120 121 122 124 126 127 128 129 130 131 132 134 136 137 138 139 140 142
 143 145 148 149]
  Test:  index=[  2   3  10  13  15  20  30  43  48  50  52  56  60  61  64  69  80  85
  89  91  92  94  95 101 106 108 111 112 116 119 123 125 133 135 141 144
 146 147]
Inner Fold 2:
  Train: index=[  2   3   7   8   9  10  13  14  15  16  18  19  20  21  22  24  25  26
  27  29  30  31  32  33  36  37  39  40  43  44  45  47  48  49  50  51
  52  54  56  57  58  59  60  61  62  63  64  65  66  67  69  70  71  72
  73  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92
  93  94  95  97  99 100 101 103 106 107 108 111 112 114 115 116 117 118
 119 121 122 123 125 126 127 130 131 132 133 134 135 136 137 138 140 141
 142 144 145 146 147]
  Test:  index=[  0   1   4   5   6  11  12  17  23  28  34  35  38  41  42  46  53  55
  68  74  75  96  98 102 104 105 109 110 113 120 124 128 129 139 143 148
 149]
Inner Fold 3:
  Train: index=[  0   1   2   3   4   5   6   7   8  10  11  12  13  15  16  17  18  20
  22  23  24  26  27  28  30  33  34  35  37  38  40  41  42  43  44  45
  46  48  50  51  52  53  54  55  56  59  60  61  62  63  64  66  68  69
  71  73  74  75  76  78  80  83  84  85  86  89  90  91  92  93  94  95
  96  97  98 100 101 102 104 105 106 107 108 109 110 111 112 113 114 116
 119 120 121 123 124 125 126 127 128 129 132 133 134 135 137 139 141 143
 144 146 147 148 149]
  Test:  index=[  9  14  19  21  25  29  31  32  36  39  47  49  57  58  65  67  70  72
  77  79  81  82  87  88  99 103 115 117 118 122 130 131 136 138 140 142
 145]
Outer Fold 0:
  Train: index=[  0   1   2   3   4   5   6   9  10  11  12  13  14  15  17  19  20  21
  23  25  28  29  30  31  32  34  35  36  38  39  41  42  43  46  47  48
  49  50  52  53  55  56  57  58  60  61  64  65  67  68  69  70  72  74
  75  77  79  80  81  82  85  87  88  89  91  92  94  95  96  98  99 101
 102 103 104 105 106 108 109 110 111 112 113 115 116 117 118 119 120 122
 123 124 125 128 129 130 131 133 135 136 138 139 140 141 142 143 144 145
 146 147 148 149]
  Test:  index=[  7   8  16  18  22  24  26  27  33  37  40  44  45  51  54  59  62  63
  66  71  73  76  78  83  84  86  90  93  97 100 107 114 121 126 127 132
 134 137]
Outer Fold 1:
  Train: index=[  0   1   4   5   6   7   8   9  11  12  14  16  17  18  19  21  22  23
  24  25  26  27  28  29  31  32  33  34  35  36  37  38  39  40  41  42
  44  45  46  47  49  51  53  54  55  57  58  59  62  63  65  66  67  68
  70  71  72  73  74  75  76  77  78  79  81  82  83  84  86  87  88  90
  93  96  97  98  99 100 102 103 104 105 107 109 110 113 114 115 117 118
 120 121 122 124 126 127 128 129 130 131 132 134 136 137 138 139 140 142
 143 145 148 149]
  Test:  index=[  2   3  10  13  15  20  30  43  48  50  52  56  60  61  64  69  80  85
  89  91  92  94  95 101 106 108 111 112 116 119 123 125 133 135 141 144
 146 147]
Outer Fold 2:
  Train: index=[  2   3   7   8   9  10  13  14  15  16  18  19  20  21  22  24  25  26
  27  29  30  31  32  33  36  37  39  40  43  44  45  47  48  49  50  51
  52  54  56  57  58  59  60  61  62  63  64  65  66  67  69  70  71  72
  73  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92
  93  94  95  97  99 100 101 103 106 107 108 111 112 114 115 116 117 118
 119 121 122 123 125 126 127 130 131 132 133 134 135 136 137 138 140 141
 142 144 145 146 147]
  Test:  index=[  0   1   4   5   6  11  12  17  23  28  34  35  38  41  42  46  53  55
  68  74  75  96  98 102 104 105 109 110 113 120 124 128 129 139 143 148
 149]
Outer Fold 3:
  Train: index=[  0   1   2   3   4   5   6   7   8  10  11  12  13  15  16  17  18  20
  22  23  24  26  27  28  30  33  34  35  37  38  40  41  42  43  44  45
  46  48  50  51  52  53  54  55  56  59  60  61  62  63  64  66  68  69
  71  73  74  75  76  78  80  83  84  85  86  89  90  91  92  93  94  95
  96  97  98 100 101 102 104 105 106 107 108 109 110 111 112 113 114 116
 119 120 121 123 124 125 126 127 128 129 132 133 134 135 137 139 141 143
 144 146 147 148 149]
  Test:  index=[  9  14  19  21  25  29  31  32  36  39  47  49  57  58  65  67  70  72
  77  79  81  82  87  88  99 103 115 117 118 122 130 131 136 138 140 142
 145]

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Nested cross-validation: why the proposed example does not have an hold-out set? #31510

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Nested cross-validation: why the proposed example does not have an hold-out set? #31510

Uh oh!

SamGG Jun 9, 2025

Replies: 2 comments

Uh oh!

Uh oh!

Mariihmp Aug 7, 2025

Uh oh!

SamGG Aug 10, 2025 Author

SamGG
Jun 9, 2025

Mariihmp
Aug 7, 2025

SamGG
Aug 10, 2025
Author