CI allow to run only selected tests but on all random seeds #23026

jeremiedbb · 2022-04-01T22:50:18Z

When using the global_random_seed fixture in new tests, we check that it works locally on all random seeds but it can happen that some seeds still fail on different platform (see #23014).

This PR gives the possibility to run the CI (azure) on only a given set of tests, for all random seeds, based on the commit message. It will trigger if the commit message is of the form:

<title> [all random seeds]
<test_name_1>
<test_name_2>
...

test_kmeans_plusplus_norms test_kmeans_elkan_results

jeremiedbb · 2022-04-01T22:57:57Z

TODO: skip azure linting (partially because next jobs depend on its success)
TODO: skip circleci

jeremiedbb · 2022-04-01T23:07:47Z

@thomasjpfan do you think it would be possible to create a workflow that triggers on some comment in the PR and pushes to the branch ?

Also do you know why the linux docker job fails because I'm just trying to add a new env var ?

thomasjpfan

@thomasjpfan do you think it would be possible to create a workflow that triggers on some comment in the PR and pushes to the branch ?

Yes. The easiest way to do it is with GitHub Actions. Although, we would need to be careful about who can activate the command tho.

thomasjpfan · 2022-04-01T23:49:53Z

build_tools/azure/posix.yml

@@ -37,6 +37,7 @@ jobs:
    TEST_DOCSTRINGS: 'false'
    CREATE_ISSUE_ON_TRACKER: 'false'
    SHOW_SHORT_SUMMARY: 'false'
+    SELECTED_TESTS: $[ dependencies.git_commit.outputs['commit.selected'] ]


Rather than having git_commit parses the commit message and pass it here, can we have a new step in poxis.yml that parses the commit message and pass it along to the next steps?

The upside is that the test configuration is responsible for parsing the commit to select tests and not the git_commit job. The downside is that we need to repeat a little bit of code for extracting the commit, but I think it's easier for future maintainers compared to using a dependencies.git_commit.

thomasjpfan · 2022-04-01T23:53:15Z

build_tools/azure/posix-docker.yml

@@ -84,6 +85,7 @@ jobs:
        -e OPENBLAS_NUM_THREADS=$OPENBLAS_NUM_THREADS
        -e SKLEARN_SKIP_NETWORK_TESTS=$SKLEARN_SKIP_NETWORK_TESTS
        -e BLAS=$BLAS
+        -e SELECTED_TESTS=$SELECTED_TESTS


Also do you know why the linux docker job fails because I'm just trying to add a new env var ?

The error is weird. My guess is that the brackets ([]) in $SELECTED_TESTS is making bash parse the command weirdly. Maybe this works:

Suggested change

-e SELECTED_TESTS=$SELECTED_TESTS

-e SELECTED_TESTS="$SELECTED_TESTS"

test_kmeans_plusplus_norms test_kmeans_elkan_results

azure-pipelines.yml

build_tools/azure/posix.yml

thomasjpfan · 2022-04-05T19:33:10Z

build_tools/azure/posix-docker.yml

+        # <test_name_2>
+        # ...
+        if [[ "$COMMIT_MESSAGE" =~ \[all\ random\ seeds\] ]]; then
+          message=$(echo $COMMIT_MESSAGE | sed -n -e 's/^.*all\ random\ seeds\] //p')


I think unix commands harder to maintain in the long run. What do you think of using a simple Python script to parse the commit message and return the selected tests?

Good idea, I moved that in a python script

test_kmeans_plusplus_norms test_kmeans_elkan_results

build_tools/azure/get_selected_tests.py

test_kmeans_plusplus_norms test_kmeans_elkan_results

build_tools/azure/get_commit_message.py

test_kmeans_plusplus_norms test_kmeans_elkan_results

thomasjpfan

LGTM

ogrisel

LGTM as well!

…earn#23026)

jeremiedbb added 2 commits April 1, 2022 23:53

Allow to run only a subset of tests but on all random seeds

a5cafe2

[all random seeds]

91a0df4

test_kmeans_plusplus_norms test_kmeans_elkan_results

github-actions bot added the Build / CI label Apr 1, 2022

thomasjpfan reviewed Apr 1, 2022

View reviewed changes

jeremiedbb added 9 commits April 5, 2022 14:26

parse commit message in templates

041edc1

tst [all random seeds]

25d7b2f

test_kmeans_plusplus_norms test_kmeans_elkan_results

new tst [all random seeds]

d87ef25

test_kmeans_plusplus_norms test_kmeans_elkan_results

new tst [all random seeds]

39f74fd

test_kmeans_plusplus_norms test_kmeans_elkan_results

new tst [all random seeds]

c018fe6

test_kmeans_plusplus_norms test_kmeans_elkan_results

debug win [all random seeds]

6383fd1

test_kmeans_plusplus_norms test_kmeans_elkan_results

tst [all random seeds]

59199a4

test_kmeans_plusplus_norms test_kmeans_elkan_results

debug win [all random seeds]

d8c3c6e

test_kmeans_plusplus_norms test_kmeans_elkan_results

tst [all random seeds]

147d540

test_kmeans_plusplus_norms test_kmeans_elkan_results

thomasjpfan reviewed Apr 5, 2022

View reviewed changes

jeremiedbb added 4 commits April 11, 2022 16:55

python script [all random seeds]

59ac79f

test_kmeans_plusplus_norms test_kmeans_elkan_results

python script [all random seeds]

772adfe

test_kmeans_plusplus_norms test_kmeans_elkan_results

python script [all random seeds]

28e545b

test_kmeans_plusplus_norms test_kmeans_elkan_results

python script [all random seeds]

156df8d

test_kmeans_plusplus_norms test_kmeans_elkan_results

jeremiedbb added the No Changelog Needed label Apr 11, 2022

jeremiedbb added 4 commits April 11, 2022 17:50

cln [all random seeds]

0fa3718

test_kmeans_plusplus_norms test_kmeans_elkan_results

exp [all random seeds]

0cbaeef

test_kmeans_plusplus_norms test_kmeans_elkan_results

cln [all random seeds]

e63c0be

test_kmeans_plusplus_norms test_kmeans_elkan_results

condition on pull requests [all random seeds]

9adbae0

test_kmeans_plusplus_norms test_kmeans_elkan_results

jeremiedbb changed the title ~~WIP CI allow to run only selected tests but on all random seeds~~ CI allow to run only selected tests but on all random seeds Apr 11, 2022

thomasjpfan mentioned this pull request Apr 11, 2022

TST Add option to run tests on 32bit data #22680

Closed

thomasjpfan reviewed Apr 11, 2022

View reviewed changes

build_tools/azure/get_selected_tests.py Show resolved Hide resolved

jeremiedbb added 2 commits April 12, 2022 12:09

exp [all random seeds]

9262c35

test_kmeans_plusplus_norms test_kmeans_elkan_results

exp [all random seeds]

7f11575

test_kmeans_plusplus_norms test_kmeans_elkan_results

jeremiedbb added 14 commits April 12, 2022 12:37

exp [all random seeds]

e0ec52d

test_kmeans_plusplus_norms test_kmeans_elkan_results

exp [all random seeds]

39d4ce1

test_kmeans_plusplus_norms test_kmeans_elkan_results

exp [all random seeds]

d9dfa52

test_kmeans_plusplus_norms test_kmeans_elkan_results

exp [all random seeds]

b6d830f

test_kmeans_plusplus_norms test_kmeans_elkan_results

cln [all random seeds]

37ce49d

test_kmeans_plusplus_norms test_kmeans_elkan_results

cln [all random seeds]

30aa92f

test_kmeans_plusplus_norms test_kmeans_elkan_results

lint [all random seeds]

576d3ab

test_kmeans_plusplus_norms test_kmeans_elkan_results

cln [all random seeds]

b180ecd

test_kmeans_plusplus_norms test_kmeans_elkan_results

cln [all random seeds]

9e4c030

test_kmeans_plusplus_norms test_kmeans_elkan_results

cln [all random seeds]

7ca5567

test_kmeans_plusplus_norms test_kmeans_elkan_results

cln [all random seeds]

0bf0b30

test_kmeans_plusplus_norms test_kmeans_elkan_results

cln [all random seeds]

2f51d13

test_kmeans_plusplus_norms test_kmeans_elkan_results

exp [all random seeds]

2df7786

test_kmeans_plusplus_norms test_kmeans_elkan_results

exp [all random seeds]

5a1f29f

test_kmeans_plusplus_norms test_kmeans_elkan_results

thomasjpfan reviewed Apr 12, 2022

View reviewed changes

build_tools/azure/get_commit_message.py Outdated Show resolved Hide resolved

build_tools/azure/get_commit_message.py Show resolved Hide resolved

jeremiedbb added 2 commits April 13, 2022 21:08

Merge branch 'master' into tst-random-seed-all

be9420b

adress comments [all random seeds]

9e36617

test_kmeans_plusplus_norms test_kmeans_elkan_results

thomasjpfan approved these changes Apr 14, 2022

View reviewed changes

ogrisel approved these changes Apr 20, 2022

View reviewed changes

ogrisel merged commit d7a0596 into scikit-learn:main Apr 20, 2022

lorentzenchr mentioned this pull request Apr 21, 2022

Investigate SAG/SAGA solver #23180

Open

This was referenced Apr 21, 2022

TST Fix ridge tests not deterministic #23181

Merged

CI Fix don't run all random seeds all the time #23190

Merged

jjerphan pushed a commit to jjerphan/scikit-learn that referenced this pull request Apr 29, 2022

CI allow to run only selected tests but on all random seeds (scikit-l…

72211d1

…earn#23026)

This was referenced Jun 23, 2022

MAINT Create private _pairwise_distances_reductions submodule #23724

Merged

Improve tests by using global_random_seed fixture to make them less seed-sensitive #22827

Open

svenstehle mentioned this pull request Jul 3, 2022

TST use global_random_seed in sklearn/linear_model/tests/test_bayes.py #23826

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI allow to run only selected tests but on all random seeds #23026

CI allow to run only selected tests but on all random seeds #23026

jeremiedbb commented Apr 1, 2022

jeremiedbb commented Apr 1, 2022

jeremiedbb commented Apr 1, 2022

thomasjpfan left a comment

thomasjpfan Apr 1, 2022

thomasjpfan Apr 1, 2022

thomasjpfan Apr 5, 2022 •

edited

Loading

jeremiedbb Apr 11, 2022

thomasjpfan left a comment

ogrisel left a comment

	-e SELECTED_TESTS=$SELECTED_TESTS
	-e SELECTED_TESTS="$SELECTED_TESTS"

CI allow to run only selected tests but on all random seeds #23026

CI allow to run only selected tests but on all random seeds #23026

Conversation

jeremiedbb commented Apr 1, 2022

jeremiedbb commented Apr 1, 2022

jeremiedbb commented Apr 1, 2022

thomasjpfan left a comment

Choose a reason for hiding this comment

thomasjpfan Apr 1, 2022

Choose a reason for hiding this comment

thomasjpfan Apr 1, 2022

Choose a reason for hiding this comment

thomasjpfan Apr 5, 2022 • edited Loading

Choose a reason for hiding this comment

jeremiedbb Apr 11, 2022

Choose a reason for hiding this comment

thomasjpfan left a comment

Choose a reason for hiding this comment

ogrisel left a comment

Choose a reason for hiding this comment

thomasjpfan Apr 5, 2022 •

edited

Loading