Skip to content

FIX Shuffle each class's samples with different random_state in StratifiedKFold #13124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Feb 27, 2019
Merged

FIX Shuffle each class's samples with different random_state in StratifiedKFold #13124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
743e73f
Enable StratifiedKFold to produce different splits
qinhanmin2014 Feb 9, 2019
a2921c6
what's new
qinhanmin2014 Feb 9, 2019
6d36012
redundant statement
qinhanmin2014 Feb 11, 2019
b7b8e44
update what's new
qinhanmin2014 Feb 11, 2019
8675188
redundant comment
qinhanmin2014 Feb 11, 2019
3bec2f1
add a test
qinhanmin2014 Feb 11, 2019
ca2f430
move what's new entry
qinhanmin2014 Feb 11, 2019
4307d64
review comment
qinhanmin2014 Feb 11, 2019
85c2d67
review comment
qinhanmin2014 Feb 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions doc/whats_new/v0.21.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,11 @@ Support for Python 3.4 and below has been officially dropped.
:func:`~model_selection.validation_curve` only the latter is required.
:issue:`12613` and :issue:`12669` by :user:`Marc Torrellas <marctorrellas>`.

- |Fix| Fixed a bug where :class:`model_selection.StratifiedKFold`
shuffles each class's samples with the same ``random_state``,
making ``shuffle=True`` ineffective.
:issue:`13124` by :user:`Hanmin Qin <qinhanmin2014>`.

:mod:`sklearn.neighbors`
........................

Expand Down
5 changes: 2 additions & 3 deletions sklearn/model_selection/_split.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -576,8 +576,7 @@ class StratifiedKFold(_BaseKFold):
``n_splits`` default value will change from 3 to 5 in v0.22.

shuffle : boolean, optional
Whether to shuffle each stratification of the data before splitting
into batches.
Whether to shuffle each class's samples before splitting into batches.

random_state : int, RandomState instance or None, optional, default=None
If int, random_state is the seed used by the random number generator;
Expand Down Expand Up @@ -620,7 +619,7 @@ def __init__(self, n_splits='warn', shuffle=False, random_state=None):
super().__init__(n_splits, shuffle, random_state)

def _make_test_folds(self, X, y=None):
rng = self.random_state
rng = check_random_state(self.random_state)
y = np.asarray(y)
type_of_target_y = type_of_target(y)
allowed_target_types = ('binary', 'multiclass')
Expand Down
11 changes: 11 additions & 0 deletions sklearn/model_selection/tests/test_split.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,17 @@ def test_shuffle_stratifiedkfold():
assert_not_equal(set(test0), set(test1))
check_cv_coverage(kf0, X_40, y, groups=None, expected_n_splits=5)

# Ensure that we shuffle each class's samples with different
# random_state in StratifiedKFold
# See https://github.com/scikit-learn/scikit-learn/pull/13124
X = np.arange(10)
y = [0] * 5 + [1] * 5
kf1 = StratifiedKFold(5, shuffle=True, random_state=0)
kf2 = StratifiedKFold(5, shuffle=True, random_state=1)
test_set1 = sorted([tuple(s[1]) for s in kf1.split(X, y)])
test_set2 = sorted([tuple(s[1]) for s in kf2.split(X, y)])
assert test_set1 != test_set2


def test_kfold_can_detect_dependent_samples_on_digits(): # see #2372
# The digits samples are dependent: they are apparently grouped by authors
Expand Down