Skip to content

DOC Add doc for permutation_test_score #17373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 31, 2020
Merged

DOC Add doc for permutation_test_score #17373

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0a256dd
amend doc string
lucyleeow May 28, 2020
cde33d1
explain high p val
lucyleeow May 28, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions sklearn/model_selection/_validation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -965,9 +965,19 @@ def _check_is_permutation(indices, n_samples):
def permutation_test_score(estimator, X, y, *, groups=None, cv=None,
n_permutations=100, n_jobs=None, random_state=0,
verbose=0, scoring=None):
"""Evaluate the significance of a cross-validated score with permutations
"""Evaluates the significance of a cross-validated score using permutations

Read more in the :ref:`User Guide <cross_validation>`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might still want to keep the link to the user manual and the end of the docstring maybe?

Copy link
Member Author

@lucyleeow lucyleeow Jun 4, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross_validation (the page linked to) never talks about permutation_test_score. In fact it is never mentioned in the user guide at all. If we wanted to add a section in the user guide it would probably fit somewhere in 'Model selection and evaluation'. I'm happy to add a section but I won't have much to say and it would be the same as what I've added in the example: #17385

Permutes targets to generate 'randomized data' and compute the empirical
p-value against the null hypothesis that features and targets are
independent.

The p-value represents the fraction of randomized data sets where the
estimator performed as well or better than in the original data. A small
p-value suggests that there is a real dependency between features and
targets which has been used by the estimator to give good predictions.
A large p-value may be due to lack of real dependency between features
and targets or the estimator was not able to use the dependency to
give good predictions.

Parameters
----------
Expand Down Expand Up @@ -1054,10 +1064,10 @@ def permutation_test_score(estimator, X, y, *, groups=None, cv=None,
-----
This function implements Test 1 in:

Ojala and Garriga. Permutation Tests for Studying Classifier
Performance. The Journal of Machine Learning Research (2010)
vol. 11
`[pdf] <http://www.jmlr.org/papers/volume11/ojala10a/ojala10a.pdf>`_.
Ojala and Garriga. `Permutation Tests for Studying Classifier
Performance
<http://www.jmlr.org/papers/volume11/ojala10a/ojala10a.pdf>`_. The
Journal of Machine Learning Research (2010) vol. 11

"""
X, y, groups = indexable(X, y, groups)
Expand Down