I think the point of this is that shuffle will always be true. we may also want to reuse KFold internally.

@jnothman Is it okay to call _iter_test_indices of KFold inside RepeatedKFold if I am inheriting RepeatedKFold from KFold?

@neerajgangwar yes, though depends on what exactly you wanna do.

I want to inherit RepeatedKFold from KFold and override _iter_test_indices which will use _iter_test_indices of KFold internally.

Calling KFold._iter_test_indices is a bit tricky unless we want to modify self.random_state which will mess with repeatability.

@jnothman do you see an easy way to reuse KFold._iter_test_indices? It also depends somewhat on what we decide in #7935, I guess?

From what I understood from the code and changes in #7935, if KFold.random_state is set to anything but int, every split call will generate different splits. So I was thinking what if we do the following RepeatedKFold.__init__

def __init__(self, n_reps=2, n_splits=3, random_state=None):
        if random_state is None:
            random_state = check_random_state(random_state)
        self.n_reps = n_reps
        super(RepeatedKFold, self).__init__(n_splits, True, random_state)

and use KFold._iter_test_indices inside RepeatedKFold._iter_test_indices since KFold._iter_test_indices shuffles the indices on every call?

We want each call to split to yield the same results, but we want each internal repetition shuffled differently. If we change self.random_state each call to split will yield a different result.

I missed that point.

@neerajgangwar well it is not true in master and the discussion in the issue I linked is somewhat complicated.

Thanks for the PR! Right now in master, it is different. But after #7935, it will not be so... So I suggest, Reinitialing a new KFold object for each repetition. Also like Andy suggests please reuse the KFold...

So I suggest, Reinitialing a new KFold object for each repetition.

It will generate different splits for each split function call.

Well you could create n_repetitions many integer random states during __init__ and then use these to create the KFold objects each time?

Yes that's an option. I was wondering if there is a better way?

In case, we create n_repetitions many integer random states and use these to create a KFold object at each repetition, RepeatedKFold can't be inherited from KFold. Would it make sense to inherit RepeatedKFold from _BaseKFold in that case?

regardless of implementation, this needs tests.

I am adding test cases and documentation.

Does that mean you should mark this MRG?

Thus RepeatedSplits(KFold(n_splits=5), n_repeats=3)

How would you control random_state for each repetition?

How would you control random_state for each repetition?

One option is to draw a random int for each instance. The other is to share a random_state and ensure they are iterated in order.

I meant once you pass an instance of KFold to RepeatedSplits (the first parameter in RepeatedSplits signature), you can't change random_state (in the current implementation).
Or did you mean that user tells RepeatedSplits or meta-splitter which cross validator instance to create and instance is created internally?

@GaelVaroquaux, @amueller who proposed the need for a RepeatedKFold: how about a RepeatedSplits or repeat_splits meta-splitter that wraps any splitter taking shuffle and random_state? Thus RepeatedSplits(KFold(n_splits=5), n_repeats=3).

I'm undecided. What other cv objects fulfil that? See http://scikit-learn.org/dev/modules/classes.html#splitter-classes

For the ShuffleSplit this doesn't really make sense, as you could just increase the number of iterations. All the others are deterministic apart from KFold and StratifiedKFold.

The main reason I wanted a class was discoverability, and because I feel these are "first class citizens" for many people.

Which option are we going ahead with? RepeatedSplits or RepeatedKFold?

_RepeatedSplits and RepeatedKFold for now? We can make _RepeatedSplits public when someone finds a use for it!

Sounds good

For this, we need set_params implemented in KFold class. Do we follow any convention for set_params and get_params methods? Any code that I can look at?

Also, I would like to submit a new PR for the new implementation. This one has a lot of unnecessary commits.

Remind me why we need set_params?

If we are going with RepeatedSplits(KFold(n_splits=5), n_repeats=3) signature then once we pass an instance of KFold to RepeatedSplits, we need to control the random_state of this instance for each repetition. This is how we are implementing it or I missed something?

Yes, but under the current setup you can just set a random_state attribute in the instance, no?

I have opened a new PR #8120. Is it okay if I close this one?