Hi @KristianGrafana! I had a quick look at the situation, and indeed that message shouldn't repeat itself for every PR. It's caused by a slightly unusual configuration pattern: the Grafana repository has multiple separate CodeQL Actions workflow files:

1. one that defines the scanning for on: push (for all languages)
2. and a bunch of others that defines the configuration for scanning on PRs (one for each language, e.g. Go)

Every single time the PR analysis runs, we check whether this CodeQL configuration already exists on the target branch. In this example: main. It concludes that that this particular code scanning configuration doesn't yet exist on main, and assumes that the PR is part of the process of setting the analysis up for pushes to main. That is how the vast majority of users go through the process of setting up code scanning. But this doesn't apply to Grafana: there already is a (separate) configuration running for pushes to main.

So on every single PR, code scanning concludes "ah, you must be in the process of setting up code scanning, and congratulations 🎉 you succeeded!".

Clearly this is not how we intend this to work for you, so we're investigating whether there's any better approach to this. Thanks for your patience!

We've temporarily switched off the "You have successfully added a ..." messages altogether while we work on a solution for this. We expect to finish work on that sometime next week, at which point new users should be seeing these messages again, but no one else. Thanks for your patience!

@sj Thank you, much appreciate the short term solution.

We've temporarily switched off the "You have successfully added a ..." messages altogether while we work on a solution for this. We expect to finish work on that sometime next week, at which point new users should be seeing these messages again, but no one else. Thanks for your patience!

We have the same issue in docker org where this message appears on every PR because code scanning result is not pushed to the default branch like docker/scout-cli#182 (comment). In our case we are using the upload-sarif action: https://github.com/github/codeql-action/blob/main/upload-sarif/action.yml. Would it be possible to have an action input to disable such message in PRs?