-
-
Notifications
You must be signed in to change notification settings - Fork 7.8k
STY: Apply pre-commit hooks to codebase #22809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
STY: Apply pre-commit hooks to codebase #22809
Conversation
.git-blame-ignore-revs
Outdated
# style: end-of-file-fixer pre-commit hook | ||
b372d66ab9e9aa30573f240d87a6ea92474cce75 | ||
|
||
# style: trailing-whitespace pre-commit hook | ||
0f0a977d8aaba773570dff8f828c358df22203aa | ||
|
||
# style: check-docstring-first pre-commit hook | ||
1a36dc83af39159749ce5857eca6ddacd345e6bb |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note to reviewer: As this file contains commits it is sensitive to rebases, and so please make sure that these commits are correct if anyone rebases this during review.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is great that this is possible. Really helpful for big cosmetic changes like this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I rebased, but I also updated the commits.
$ git log --graph --oneline --decorate --all | head -n 10
* 02fc0611cf (HEAD -> mnt/apply-pre-commit-to-all-files, origin/mnt/apply-pre-commit-to-all-files) DOC: Add pre-commit hook run dev instructions
* f3aa826d83 DEV: Add .git-blame-ignore-revs to ignore style in blame view
* 66c12f2778 STY: Apply check-docstring-first pre-commit hook
* 33516082c5 STY: Apply trailing-whitespace pre-commit hook
* 94d985a744 STY: Apply end-of-file-fixer pre-commit hook
* 7e04925fcd MNT: Update pre-commit hooks to latest versions
| * 3dd6049160 (mnt/add-codespell-to-pre-commit-hooks) FIX: Fix typo caught by codespell pre-commit hook
| * 19987645ec DEV: Add codespell to pre-commit hooks
|/
* 19992285cb (upstream/main, origin/main, main) Merge pull request #22807 from anntzer/qd
The hooks can also be run manually. All the hooks can be run, in order as | ||
listed in ``.pre-commit-config.yaml``, against the full codebase with :: | ||
|
||
pre-commit run --all-files | ||
|
||
To run a particular hook manually, run ``pre-commit run`` with the hook id :: | ||
|
||
pre-commit run <hook id> --all-files |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can of course target particular files with --file
(e.g. pre-commit run --file README.rst
) but given that is already done on the files that you're attempting to commit when you have pre-commit installed locally this didn't really seem important to note.
From pre-commit run --help
:
--files [FILES ...] Specific filenames to run hooks on.
To address a question from Gitter: In PR #17143 matplotlib/.pre-commit-config.yaml Line 3 in a436253
open a PR to automatically update the hooks that have new releases. It also runs all of the pre-commit hooks faster then I'll leave it up to the dev team if this is sufficiently interesting/useful to have. |
I also enabled pre-commit.ci on my fork to do a demo of what this looks like. c.f. matthewfeickert#9 which shows things working, and then also a little autofix demo from a comment. |
I'm not sure that we should require pre-commit hooks (as I take is an indirect suggestion here). While I tend to use it, there are setups where it is not really convenient. For example, I have different shells on Windows, one for running git (https://gitforwindows.org/) and one for Python (conda). While I can of course unify them (being too lazy so far, and doing much development on Linux), this is a bit of extra work and nothing I think should be an obstacle for contributing. Also, I am not sure how this works when doing commits from graphical interfaces (which for some reason seems to be a popular way of using git...). Running the hooks on all files is as you say a one-time effort. IMHO I do not think we need to use the To me, pre-commit hooks primarily serve the purpose of not having to run the code quality tests locally and get warnings if something is not right, so I am not a big fan of the hooks actually changing stuff. Better to "learn" how it should be done. |
This doesn't require developers to use them. Using pre-commit.ci is just going to find things that would later require a developer to go fix to and alert the PR contributor and the reviewers to them.
I don't think I understand. :? What is the obstacle? Are you saying that
That is an interesting point that I've never thought about before. I would be a bit surprised (maybe naively?) to find out that popular Git GUIs have no plugin support though, and so I would (definitely naively) hope things "just work".
It is certainly not required, but the diff is pretty ugly and meaningless, so this was more of a "making it less painful" effort. So I'm quite happy to remove this, but the blame avoidance seems almost custom designed for these huge diffs situations.
I take a different view on their purpose (automate the boring stuff that would need to get human changed anyway), but my opinion on that isn't the relevant thing here. The hooks only run locally if the developer has them installed locally (which is totally optional) and if a developer pushes code that fails pre-commit.ci then as automatic pushing back fixes is disabled matplotlib/.pre-commit-config.yaml Line 2 in a436253
they can speed up the process and avoid having a reviewer have to ask for them for changes by just commenting "pre-commit.ci autofix". Nothing is happening automatically unless a developer opts in by using pre-commit locally or by asking pre-commit.ci to fix things for them. (I think I am probably telling you things that you already know though, so apologies here. 😬 Can you elaborate/clarify your points if I've misunderstood?) I'll note also that this PR is simply enforcing the pre-commit hooks that were added in PR #21583, and if the dev team isn't happy with that hook selection I'm certainly happy to fix that here to whatever is desired. 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the git-blame-ignore-revs, I think these cosmetic changes are fine to get the recommit hooks to pass.
The problem is that the git (for Windows) bash shell and the (mini)conda shell (Anaconda prompt) don't have the same Python environment, so if I run git in the conda shell I have everything I need for the pre-commit, but not the nice aspects of the git bash shell (like branch name and tab-completion), and if I use the git shell, I do not have the conda environment. As I said, this can be sorted out with a bit of effort and is just a convenience thing really, but I would assume it is a similar situation for the graphical clients as conda typically doesn't set up any environment variables except for it's own shell. (I think I have three different Python installs on my Windows machine: conda, that I use for most things, Windows Python that ends up in the path, and the one provided with the MS dev-prompt for building certain stuff. So while I can have all three synced, it is not very practical...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm actually surprised there is this little churn in the files, it is almost all trailing whitespace.
I also prefer having the .git-blame-ignore-revs
for the whitespace changes because what I'm really interested in with the blame is looking for who updated the actual code piece and I don't want to ping @matthewfeickert to fix something he only did a whitespace update on :)
lib/matplotlib/backend_tools.py
Outdated
@@ -989,12 +989,12 @@ def trigger(self, *args, **kwargs): | |||
'help': ToolHelpBase, | |||
'copy': ToolCopyToClipboardBase, | |||
} | |||
"""Default tools""" | |||
# Default tools |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just delete this comment and the one below, the comment is the same as the variable name :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in rebase I just pushed. 👍
c677636
to
02fc061
Compare
Just showing that things still pass after rebase: $ pre-commit run --all-files
check for added large files..............................................Passed
check docstring is first.................................................Passed
fix end of files.........................................................Passed
mixed line ending........................................................Passed
trim trailing whitespace.................................................Passed
flake8...................................................................Passed |
@@ -989,12 +989,10 @@ def trigger(self, *args, **kwargs): | |||
'help': ToolHelpBase, | |||
'copy': ToolCopyToClipboardBase, | |||
} | |||
"""Default tools""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are these removed? Not sure if they are actually rendered, but those are doc-strings for the attributes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They were removed given @greglucas review comment here: #22809 (comment)
They don't show up in the "files changed" tab during review and only show up in the "Conversation" tab because I rebased the PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK! Yes, considering that the doc-string is the same as the name and that they do not show up in the rendered documentation it may make sense. (Not having them as a comment for sure makes sense.)
I was a bit worried that this indicates that one cannot really have variable doc-strings though. That those will be interpreted as "multiple top-level doc-strings".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I was wrong. This change leads to a modification of the documentation. If that is wanted or not, I cannot really tell, but I do not think it is good if it was enforced by some pre-commit setting.
Current:
After this PR it is removed:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@oscargus I'll let you and the other members of the dev team make a call here, but the fact that this does show up in the current docs looks unintentional to me given the render. At least I don't see their value here in the docs, but I'm also obviously not an expert in this section of the API.
These docstring comments were added in 2015 back in PR #3652 by @fariza. So while I sure hope no one ever asks me to explain code that I wrote over 5 years ago 😬 maybe they can take a look and give their thoughts.
@greglucas as you suggested removal maybe you can also give your thoughts here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI, check-docstring-first had a long-standing issue for these false positives, see pre-commit/pre-commit-hooks#159. It can be turned off per-file, and does catch the common case where imports sneak above the docstring. Or it can be dropped, either way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this affects excess docstrings in examples far more than variable docstrings in the actual library, so I think I'm net positive on keeping it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so I think I'm net positive on keeping it.
(Sorry, long week and my brain is a bit fuzzy.) Positive on keeping the check-docstring-first
hook, correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm pretty indifferent on this (as it sounds like most of us are!). I'd lean towards sticking with what you have currently (removing the docstring/comment [and because of that removing the sphinx generated documentation] and leaving the pre-commit check in place).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@QuLogic @oscargus @jklymak @greglucas @tacaswell As it seems that there are no strong feelings against keeping the check-docstring-first
hook and some positive leanings towards it, does anyone have any further questions on this PR?
To keep the noise down on this I've stopped rebasing it against main
but I have a branch on my fork where I've continued to do so and pre-commit run --all-files
is still passing with this rebased on main
's HEAD
.
That is the general reason, yes. However, looking at the changes here: do you really expect that you will have to ping for any of these changes? They are primarily in non-code and the very few that are in the code are quite non-critical. |
I agree that this specific change isn't too invasive and wouldn't be a big issue either way for the ignore commits from my perspective. However, I see this as setting a good precedent for potential future updates dropping big style changes and makes it easy to see how to do it properly by just adding the commit hash to this file. |
Yeah. 👍 I got the idea from @henryiii using it in scikit-build/scikit-build#685 where they use Black. I know that Matplotlib doesn't use any formatters like that, but if there was ever a big style revision it could be done with a diminished disruption with respect to using the Git history. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a slight preference that the LICENCES files not be touched / be ignored by these tools but not a big enough concern to block over it.
@tacaswell That can easily happen. They can just get added to the exclude list matplotlib/.pre-commit-config.yaml Lines 4 to 12 in c33557d
I'll do that, rebase to get the right commits, and then request re-review. |
Requested by Thomas Caswell in PR 22809
* Update pre-commit hooks: - https://github.com/pre-commit/pre-commit-hooks: v4.0.1 → v4.2.0 - https://github.com/pycqa/flake8: v3.9.2 → v4.0.1 This is being applied manually and not waiting for pre-commit.ci to apply it as pre-commit.ci hasn't been turned on yet since these hooks were introduced in PR 21583.
Also remove comments in lib/matplotlib/backend_tools.py per request of maintainer in PR review.
Add a git blame ignore-revs file to automatically ignore commits listed in it from the blame view on GitHub. Include in it style changes added by pre-commit hooks which touch multiple files. To ignore the commits in the .git-blame-ignore-revs when running `git blame` locally run: `git blame --ignore-revs-file .git-blame-ignore-revs`. c.f.: * https://git-scm.com/docs/git-blame#Documentation/git-blame.txt---ignore-revs-fileltfilegt * https://docs.github.com/en/repositories/working-with-files/using-files/viewing-a-file#ignore-commits-in-the-blame-view
Add instructions to the devs on how to manually run pre-commit hooks, both for all the hooks, and for individual hook ids. Example: `pre-commit run flake8 --all-files` Also use the 'python -m pip' pattern to be consistent with the rest of the docs.
02fc061
to
48ade7d
Compare
# style: end-of-file-fixer pre-commit hook | ||
c1a33a481b9c2df605bcb9bef9c19fe65c3dac21 | ||
|
||
# style: trailing-whitespace pre-commit hook | ||
213061c0804530d04bbbd5c259f10dc8504e5b2b | ||
|
||
# style: check-docstring-first pre-commit hook | ||
046533797725293dfc2a6edb9f536b25f08aa636 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After this last rebase:
$ git log --graph --oneline --decorate --all | head -n 7
* 48ade7d904 (HEAD -> mnt/apply-pre-commit-to-all-files, origin/mnt/apply-pre-commit-to-all-files) DOC: Add pre-commit hook run dev instructions
* 4c003775bb DEV: Add .git-blame-ignore-revs to ignore style in blame view
* 0465337977 STY: Apply check-docstring-first pre-commit hook
* 213061c080 STY: Apply trailing-whitespace pre-commit hook
* c1a33a481b STY: Apply end-of-file-fixer pre-commit hook
* 787bdf08e5 MNT: Update pre-commit hooks to latest versions
* 95038d96a4 DEV: Exclude LICENSE files from pre-commit
and
$ git grep "c1a33a481b\|0465337977\|213061c080"
.git-blame-ignore-revs:c1a33a481b9c2df605bcb9bef9c19fe65c3dac21
.git-blame-ignore-revs:213061c0804530d04bbbd5c259f10dc8504e5b2b
.git-blame-ignore-revs:046533797725293dfc2a6edb9f536b25f08aa636
so these are correct. 👍
As a single line checker:
$ git log --oneline | head -n 7 | grep STY | awk '{print $1}' | xargs -I {} git grep {}
.git-blame-ignore-revs:046533797725293dfc2a6edb9f536b25f08aa636
.git-blame-ignore-revs:213061c0804530d04bbbd5c259f10dc8504e5b2b
.git-blame-ignore-revs:c1a33a481b9c2df605bcb9bef9c19fe65c3dac21
@tacaswell The $ pre-commit run --all-files
check for added large files..............................................Passed
check docstring is first.................................................Passed
fix end of files.........................................................Passed
mixed line ending........................................................Passed
trim trailing whitespace.................................................Passed
flake8...................................................................Passed
$ git diff upstream/main -- LICENSE/ # No diff
$ |
Fair enough! |
Thanks @matthewfeickert ! |
PR Summary
This is a follow up PR to PR #21583 based on discussions in PR #22777 (c.f. #22777 (comment)) and on Gitter. This touches a lot of files so I'll walk through the story here, which should help motivate turning on pre-commit.ci.
matplotlib/.pre-commit-config.yaml
Line 3 in a436253
though as pre-commit.ci wasn't turned on, there were no pre-commit hook updates coming in on that schedule and so I've run
pre-commit autoupdate
to update thepre-commit-hooks
andflake8
ids.With those updated, I then ran the failing (c.f. DEV: Add codespell to pre-commit hooks #22777 (comment))
pre-commit
hooks and applied their necessary revisions as individual commits. So those are the style commits forend-of-file-fixer
,trailing-whitespace
, andcheck-docstring-first
. These were also done as individual commits so that if in review any of these are viewed as not acceptable they can be dropped easily with a rebase.As the following commits touch many files, this is a noisy first time run of the hooks. To help limit this, add a
.git-blame-ignore-revs
file that will automatically ignore commits in the blame view on GitHub (this file is required to be in the top level of the repo). This can also be done locally withgit blame
by runningExample on my fork:

Note to reviewer: As this file contains commits it is sensitive to rebases, and so please make sure that these commits are correct if anyone rebases this during review.
This PR should probably be merged in advance of PR #22777 which can be trivially rebased and would be a good time to turn pre-commit.ci on.
pre-commit
now passes the full repo:PR Checklist
Tests and Styling
pytest
passes).flake8-docstrings
and runflake8 --docstring-convention=all
).Documentation
doc/users/next_whats_new/
(follow instructions in README.rst there).doc/api/next_api_changes/
(follow instructions in README.rst there).