MAINT Adds script to create or update issue when CI fails #21544

thomasjpfan · 2021-11-03T18:16:56Z

Reference Issues/PRs

Related to #21286

What does this implement/fix? Explain your changes.

This PR automatically creates an issue or update an existing issue when a scheduled CI fails. Here is what it looks like: thomasjpfan#92

Any other comments?

The script is CI agnostic and should be easy to use with GitHub Actions.

CC @ogrisel

thomasjpfan · 2021-11-03T18:17:57Z

maint_tools/create_issue_from_juint.py

+    # TODO: Do we want to close the issue here?
+    print("Test has no failures!")


If there are not failures should we automatically close the issue if it exist?

I think we could. If not we should at least update any existing issue.

I don't know, what would be the false positive rate on auto close?

ogrisel · 2021-11-04T10:03:39Z

/cc @jjerphan as you might be interested in an example of usage of the GitHub API.

ogrisel · 2021-11-04T10:06:25Z

The report is really great. Maybe we could generalize this idea of analyzing the CI output by maintaining a high level, user friendly short summary of the main problems of a given PRs (e.g. at the bottom of the PR description or in a continuously updated comment maintained by the bot) without having to go through the complex Azure Pipelines UI and verbose logs.

ogrisel · 2021-11-04T10:10:40Z

Could you please also update the CI section of the maintainers doc to mention what this bots does? In particular if you need particular account credentials it would be great to make it explicit how those are managed and where the secret tokens are configured (if any).

glemaitre · 2021-11-04T15:52:24Z

Maybe we could generalize this idea of analyzing the CI output by maintaining a high level, user-friendly short summary of the main problems of a given PRs

I am +1 for this feature. I really find the clicking pattern to go to the CI quite intimidating for newcomers.

adrinjalali · 2021-11-06T15:01:26Z

azure-pipelines.yml

@@ -144,6 +144,7 @@ jobs:
        BLAS: 'openblas'
        COVERAGE: 'false'
        BUILD_WITH_ICC: 'false'
+        CREATE_ISSUE_ON_TRACKER: 'true'


agreed to move this to the upstream one

jjerphan

LGTM. Thanks @thomasjpfan for setting this up!

maint_tools/create_issue_from_juint.py

adrinjalali · 2021-11-08T15:17:27Z

build_tools/azure/posix.yml

+        versionSpec: '3.9'
+      displayName: Place Python into path to update issue tracker
+      condition: and(succeededOrFailed(), eq(variables['CREATE_ISSUE_ON_TRACKER'], 'true'),
+                     in(variables['Build.Reason'], 'Schedule', 'IndividualCI'))


does this mean we run this for individual CI runs? I thought we don't want that, since we only have this for nightly scipy-dev runs

adrinjalali · 2021-11-08T15:32:47Z

maint_tools/create_issue_from_juint.py

+title = f"⚠️ CI failed on {args.ci_name} ⚠️"
+
+
+def get_issue():


would it be possible to check the dependency and scikit-learn versions of the open issue against the one in the current CI, and then only return that issue if they match?

It is possible, but I think it makes everything a bit more complicated. If we check the dependencies and scikit-learn version, then the scipy-dev config may have multiple open issues. For example, if the scipy-dev config has updated dependencies, it would open another issue. This happens quite frequently because scipy-dev uses the nightly builds. In these cases, we either let the CI close the old issues, or we close it ourselves.

I prefer the current implementation since it has a one-to-one mapping between a [scipy-dev] and an open issue.

If [scipy-dev] is passing -> close issue connected to [scipy-dev]

If [scipy-dev] is failing -> update issue connected to [scipy-dev] or create a new issue if there isn't an open issue connected to [scipy-dev]

Fair enough, we can let it be and see how it works. I'd also be happy if we had a ping to @core-devs to the first post of each issue. We can then unsubscribe if we want.

maint_tools/create_issue_from_juint.py

adrinjalali · 2021-11-08T15:37:54Z

maint_tools/create_issue_from_juint.py

+    print("Test has no failures!")
+    issue = get_issue()
+    if issue is not None:
+        print(f"Closing issue #{issue.number}")


I think this is where the version check would be useful, so that we only close automatically if the old one used the same versions

adrinjalali · 2021-11-08T16:11:33Z

I'll let @ogrisel do another check and merge if all good.

jjerphan · 2021-11-08T16:13:52Z

IIRC, @ogrisel is off this week.

ogrisel

LGTM, let's give it a shot. I will then do a follow-up PR to move the PyPy job as a scheduled job with this option enabled.

…rn#21544)

MAINT Adds script to create or update issue when CI fails

0b7766a

github-actions bot added the Build / CI label Nov 3, 2021

thomasjpfan commented Nov 3, 2021

View reviewed changes

jjerphan self-requested a review November 4, 2021 10:14

adrinjalali reviewed Nov 6, 2021

View reviewed changes

thomasjpfan added 5 commits November 6, 2021 18:13

Merge remote-tracking branch 'upstream/main' into upstream_github_issue

dc90b3d

ENH Auto close issue

e59fc9c

CLN Nicer UI

931ab34

DOC Describe token requirements

ffdebee

DOC Describe token requirements

8b27977

jjerphan approved these changes Nov 7, 2021

View reviewed changes

maint_tools/create_issue_from_juint.py Outdated Show resolved Hide resolved

CLN Address comments

00e5c2d

adrinjalali reviewed Nov 8, 2021

View reviewed changes

thomasjpfan added 2 commits November 8, 2021 10:52

Merge remote-tracking branch 'upstream/main' into upstream_github_issue

d46c6bb

CLN Address comments

249d735

adrinjalali approved these changes Nov 8, 2021

View reviewed changes

ogrisel approved these changes Nov 12, 2021

View reviewed changes

ogrisel merged commit 2ee1305 into scikit-learn:main Nov 12, 2021

ogrisel mentioned this pull request Nov 12, 2021

MAINT move PyPy back to scheduled build #21645

Merged

glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Nov 22, 2021

MAINT Adds script to create or update issue when CI fails (scikit-lea…

a9d8c1e

…rn#21544)

glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Nov 29, 2021

MAINT Adds script to create or update issue when CI fails (scikit-lea…

d1e2358

…rn#21544)

samronsin pushed a commit to samronsin/scikit-learn that referenced this pull request Nov 30, 2021

MAINT Adds script to create or update issue when CI fails (scikit-lea…

51e20c3

…rn#21544)

glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Dec 24, 2021

MAINT Adds script to create or update issue when CI fails (scikit-lea…

ebd21e2

…rn#21544)

glemaitre pushed a commit that referenced this pull request Dec 25, 2021

MAINT Adds script to create or update issue when CI fails (#21544)

716d16a

		# TODO: Do we want to close the issue here?
		print("Test has no failures!")

		title = f"⚠️ CI failed on {args.ci_name} ⚠️"


		def get_issue():

Uh oh!

MAINT Adds script to create or update issue when CI fails #21544

MAINT Adds script to create or update issue when CI fails #21544

Uh oh!

Conversation

thomasjpfan commented Nov 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Nov 4, 2021

Uh oh!

ogrisel commented Nov 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ogrisel commented Nov 4, 2021

Uh oh!

glemaitre commented Nov 4, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjerphan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adrinjalali commented Nov 8, 2021

Uh oh!

jjerphan commented Nov 8, 2021

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thomasjpfan commented Nov 3, 2021 •

edited

Loading

ogrisel commented Nov 4, 2021 •

edited

Loading