[docs] Strengthen our quality standards and connect AI contribution policy to it #154441

rnk · 2025-08-19T23:42:36Z

Open source projects are often the target of nuisance contributions (think typo corrections to comments in test cases), but prior to the advent of AI tools, we hadn't had to take a strong policy stance discouraging these contributions. However, as the tools have improved, the skill required to make plausible contributions has decreased, and burdensome nuisance contributions have become a problem for our project, so I think we should adopt a stronger quality bar policy.

I've updated our existing Quality section to frame everything in terms of a golden rule that contributions should pass subjective maintainer quality standards, held in balance against our desire to welcome new contributors.

I then connected the AI policy section to our core quality policy, to try to make the AI policy section tool-neutral. My goal is to minimize the distinction between traditional mass-refactoring tools, and the costs of those migrations, and AI-assisted contributions.

This is following up on a discussion in the July 17 LLVM project council meeting, and while I could probably keep editing this forever, I'm posting this now so we can discuss it in the project council meeting tomorrow.

…olicy to it Open source projects are often the target of nuisance contributions (think typo corrections), but prior to the advent of AI tools, we hadn't had to take a strong policy stance discouraging these contributions. However, as the tools have improved, the skill required to make plausible contributions has decreased, and burdensome nuisance contributions have become a problem for our project, so I think we should adopt a stronger quality bar policy. I've updated our existing Quality section to frame everything in terms of a golden rule that contributions should pass subjective maintainer quality standards, held in balance against our desire to welcome new contributors. I then connected the AI policy section to our core quality policy, to try to make the AI policy section tool-neutral. My goal is to minimize the distinction between traditional mass-refactoring tools, and the costs of those migrations, and AI-assisted contributions. This is following up on a discussion in the July 17 LLVM project council meeting.

boomanaiden154 · 2025-08-19T23:57:29Z

llvm/docs/DeveloperPolicy.rst

-   where "reasonable" depends on the contributor's judgement and the scope of
-   the change (more invasive changes require more testing). A reasonable subset
-   might be something like "``llvm-test/MultiSource/Benchmarks``".
+#. Code must pass the relevant regression test suites on all supported


The wording here doesn't convey (to me) what I think you intend.

LLVM doesn't really have a good concept of what a "supported platform" is. In the limit, this can be anything there is a buildbot for. Even people who have been contributing for a while (or at least me) regularly break more esoteric platforms. I think something like passing premerge checks better captures the intention here because it's well defined in the community and something a contributor can actually iterate on (unlike ensuring your code works on a SPARC Solaris machine).

Your section on postsubmit responsibilities seems to capture this pretty well though.

LLVM doesn't really have a good concept of what a "supported platform" is.

I guess I think we should define LLVM-wide project support tiers similar to what Rust and libc++ have, but until we have those, I agree, it's more helpful to rephrase this in terms of passing premerge build & test, and then the policy becomes evergreen, defined by our CI config.

boomanaiden154 · 2025-08-20T00:04:33Z

llvm/docs/DeveloperPolicy.rst

+attentional resources of LLVM project maintainers, the onus is on contributors
+to justify why their contributions are not burdensome and exceed our `quality`_
+bar. Contributors who repeatedly send low-quality contributions to our project
+will be subject to escalating moderation actions and eventually a project ban.


Is there a formal process for how a ban would be decided?

I know in the past it has gone through the code of conduct committee. I think it's important to explicitly signal that these cases are decided through a fair process rather than just handed out without any investigation.

Maybe we should explicitly state that the CoC committee (or maybe area teams) can place temporary restrictions on a person making pull requests or other contributions. Like, a limit of one open pull request at a time, or no AI-generated bug reports, or whatever we think is appropriate based on the pattern. And then violating a temporary restriction is a CoC violation which can lead to a general ban. (So we wouldn't directly ban people under this policy for low-quality contributions; we would only ban people for violating a temporary restriction.)

The CoC committee did this once before (https://discourse.llvm.org/t/llvm-code-of-conduct-transparency-report-july-15-2023-july-15-2024/82687#p-332791-report-3-a-user-is-submitting-a-very-large-number-of-low-quality-pull-requests-17), although it wasn't backed by any formal policy at the time.

A more formal process would be good, but I wanted to keep this high-level and empower our existing space moderators on GitHub, Discourse, Discord, to take these actions using their own discretion, rather than trying to hammer out objective, crystal-clear rules in this policy update.

llvm/docs/DeveloperPolicy.rst

cmtice · 2025-08-20T15:55:37Z

llvm/docs/DeveloperPolicy.rst

+has increased the asymmetry between the work of producing a contribution, and
+the work of reviewing the contribution. In order to protect the time and
+attentional resources of LLVM project maintainers, the onus is on contributors
+to justify why their contributions are not burdensome and exceed our `quality`_


Do they need to exceed it or just meet it?

I implemented this wording change

dwblaikie · 2025-08-20T16:03:06Z

llvm/docs/DeveloperPolicy.rst

+
+* The code should compile cleanly on all supported platforms.
+
+* The changes should not cause any correctness regressions in the ``llvm-test``


I guess this is refering to https://github.com/llvm/llvm-test-suite ? (so maybe call it llvm-test-suite & link to the repo?

This was the text as it was written in the previous pre-commit quality bar section, but yes, I interpret it that way, and I'll make that update.

…ions

nikic

Some high level feedback:

I think this is a bit too verbose for a policy document. Some of this (like the references to other AI policies) are something to mention in an RFC to justify how we arrived here, but they are not something every person reading the finished policy needs to look at. Similarly, I'm not sure we should be including the meta philosophy around "extractive" contributions in the policy document. We should be able to explain this more concisely -- is the contribution worth the required investment of reviewer time?
One of the motivations behind revisiting this policy is to avoid wasting reviewer time on low-quality AI generated PRs. I think what I'm currently missing from this policy is a means by which a reviewer can reject a low-quality PR without significant justification. Basically, this needs to be at the level of rejecting a PR with a copy&paste comment pointing to the policy.

nikic · 2025-08-21T10:26:28Z

llvm/docs/DeveloperPolicy.rst

+the LLVM project to be welcoming and open to aspiring compiler engineers who are
+willing to invest time and effort to learn and grow, because growing our
+contributor base and recruiting new maintainers helps sustain the project over
+the long term. We therefore label pull requests from new contributors and


We do? What's the label for that?

I based this on glancing at this github workflow:
https://github.com/llvm/llvm-project/blob/main/.github/workflows/new-prs.yml#L24

On second glance, we don't have a label, but we do post a greeting. I'll just update the text to match reality, but I think a label would be helpful too, since it can be used to specifically focus on reviewing PRs from new contributors. I realize I'm speaking as someone who only reviews things sent to me, but this seems like a good aspirational goal.

nikic · 2025-08-21T10:29:12Z

llvm/docs/DeveloperPolicy.rst

+We prefer for this to be handled before submission but understand that it isn't
+possible to test all of this for every submission.  Our build bots and nightly
+testing infrastructure normally finds these problems.  A good rule of thumb is
+to check the nightly testers for regressions the day after your change.  Build


What nightly testers?

This is stale text, but the text should capture any tests that we cannot afford to run continuously, and that typically includes compile time regression tracking (llvm-compile-time-tracker.com) and performance testing (lnt.llvm.org, yes, it's down).

rnk

I had pending comments, sorry these are stale, I wrote them last week. It makes more sense for me to send these as one update, and then focus on new feedback post project council meeting.

rnk · 2025-08-20T16:36:12Z

llvm/docs/DeveloperPolicy.rst

-   where "reasonable" depends on the contributor's judgement and the scope of
-   the change (more invasive changes require more testing). A reasonable subset
-   might be something like "``llvm-test/MultiSource/Benchmarks``".
+#. Code must pass the relevant regression test suites on all supported


LLVM doesn't really have a good concept of what a "supported platform" is.

I guess I think we should define LLVM-wide project support tiers similar to what Rust and libc++ have, but until we have those, I agree, it's more helpful to rephrase this in terms of passing premerge build & test, and then the policy becomes evergreen, defined by our CI config.

llvm/docs/DeveloperPolicy.rst

rnk · 2025-08-20T16:39:34Z

llvm/docs/DeveloperPolicy.rst

+
+* The code should compile cleanly on all supported platforms.
+
+* The changes should not cause any correctness regressions in the ``llvm-test``


This was the text as it was written in the previous pre-commit quality bar section, but yes, I interpret it that way, and I'll make that update.

rnk · 2025-08-20T16:44:47Z

llvm/docs/DeveloperPolicy.rst

+attentional resources of LLVM project maintainers, the onus is on contributors
+to justify why their contributions are not burdensome and exceed our `quality`_
+bar. Contributors who repeatedly send low-quality contributions to our project
+will be subject to escalating moderation actions and eventually a project ban.


A more formal process would be good, but I wanted to keep this high-level and empower our existing space moderators on GitHub, Discourse, Discord, to take these actions using their own discretion, rather than trying to hammer out objective, crystal-clear rules in this policy update.

rnk · 2025-08-25T21:45:33Z

Some high level feedback:

I think this is a bit too verbose for a policy document. Some of this (like the references to other AI policies) are something to mention in an RFC to justify how we arrived here, but they are not something every person reading the finished policy needs to look at. Similarly, I'm not sure we should be including the meta philosophy around "extractive" contributions in the policy document. We should be able to explain this more concisely -- is the contribution worth the required investment of reviewer time?

My view is that these references are a good way of capturing the "alternatives considered", which is often the most important part of any technical design document. They are also likely to be helpful in capturing the "why" of any policy, which is something that the next policy doc editor will probably care about (I now find myself in this position). I've personally received a lot of links to what other projects are doing, and so the section serves as a way of saying "yes, we looked at all that, and decided we want to make compromise X".

I think the Developer Policy doc is just growing, and it needs to be split up (LangRef has grown similarly), and it also needs to be adapted to apply to the entire LLVM umbrella project. Much of the text is focused on LLVM the optimizer and code generator.

Re: extractive contributions, @chandlerc gave me the opposite feedback, that I should center the ideas of extractive and non-extractive contributions. I defer to the opinions of active maintainers, but that's why I went philosophical. Quality is inherently subjective and philosophical, at the end of the ady.

One of the motivations behind revisiting this policy is to avoid wasting reviewer time on low-quality AI generated PRs. I think what I'm currently missing from this policy is a means by which a reviewer can reject a low-quality PR without significant justification. Basically, this needs to be at the level of rejecting a PR with a copy&paste comment pointing to the policy.

This is true, I will take action on that, I got the same feedback from Aaron. I do need to get that PR-copy-pasta message into the proposal near the top.

boomanaiden154 reviewed Aug 20, 2025

View reviewed changes

chandlerc reviewed Aug 20, 2025

View reviewed changes

llvm/docs/DeveloperPolicy.rst Outdated Show resolved Hide resolved

llvm/docs/DeveloperPolicy.rst Outdated Show resolved Hide resolved

llvm/docs/DeveloperPolicy.rst Outdated Show resolved Hide resolved

cmtice reviewed Aug 20, 2025

View reviewed changes

dwblaikie reviewed Aug 20, 2025

View reviewed changes

Updating text to center it around extractive/non-extractive contribut…

8f861a0

…ions

nikic reviewed Aug 21, 2025

View reviewed changes

rnk commented Aug 25, 2025

View reviewed changes

rnk added 2 commits August 25, 2025 14:57

Address feedback on nightly testing and add copy-pasted message

c52546a

Update wording: We comment on first-time PRs, we do not label them (yet)

dd6e729


		* The code should compile cleanly on all supported platforms.

		* The changes should not cause any correctness regressions in the ``llvm-test``

[docs] Strengthen our quality standards and connect AI contribution policy to it #154441

Are you sure you want to change the base?

[docs] Strengthen our quality standards and connect AI contribution policy to it #154441

Uh oh!

Conversation

rnk commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rnk Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rnk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rnk commented Aug 25, 2025

Uh oh!

Uh oh!

rnk commented Aug 19, 2025 •

edited

Loading

rnk Aug 25, 2025 •

edited

Loading