Are the pyright configs used when checking scripts? As far as I understand, our scripts are currently checked with mypy only, at least in CI. That said, I'd be glad if we could apply this PR.

Cc @AlexWaygood

The pyright configs are not used when checking our scripts, but they are used when checking our test-case files. The stdlib test cases have a dedicated pyrightconfig file all to themselves, but the third-party test cases use the same config files as our third-party stubs (they have to, because the third-party test cases are in the same top-level folder as the third-party stubs).

The reason why I initially introduced this check into check_consistent.py was because we didn't at the time have reportUnnecessaryTypeIgnoreComment=true enabled for the stubs/ directory. But it's crucial to have this setting enabled for our test cases, or pyright won't verify our test cases that check for false negatives. Our tests against false negatives rely on us using type: ignore comments on invalid code; as long as type checkers don't emit "unused type-ignore comment" errors on these lines, we know the stubs are working as intended.

Since, as @Avasam says, reportUnnecessaryTypeIgnoreComment=true is nowadays enabled for the whole of the stubs directory, we can remove this check from check_consistent.py. But this makes me realise that our pyright settings for our test cases are probably now broken in a new way. This line in our pyright config files means that pyright now straight-up ignores type: ignore comments altogether in the stubs/ directory:

Presumably pyright won't emit any errors for unused type: ignore comments if it's just ignoring all of them anyway. So maybe we should replace the current check that enforces reportUnnecessaryTypeIgnoreComment=true to be added at the top of each third-party test-case file with a new check that enforces enableTypeIgnoreComments=true to be added at the top of each third-party test-case file.

Are the pyright configs used when checking scripts? As far as I understand, our scripts are currently checked with mypy only, at least in CI. That said, I'd be glad if we could apply this PR.

Pyright is run on the test_cases folders when run with the pyrightconfig.testcases.json configuration.

This line in our pyright config files means that pyright now straight-up ignores type: ignore comments altogether in the stubs/ directory:
[...]
Presumably pyright won't emit any errors for unused type: ignore comments if it's just ignoring all of them anyway.

@AlexWaygood "enableTypeIgnoreComments": true specifically for test_cases. So it doesn't ignore it, with a comment to warn anyone who'd want to deactivate it:

python .\tests\pyright_test.py .\stubs\SQLAlchemy\@tests -p .\pyrightconfig.testcases.json

8.19.2
Running: C:\Program Files\nodejs\npx.CMD pyright@1.1.285 .\stubs\SQLAlchemy\@tests -p .\pyrightconfig.testcases.json
Loading configuration file at E:\Users\Avasam\Documents\Git\typeshed\pyrightconfig.testcases.json
Assuming Python version 3.9
Assuming Python platform Windows
Auto-excluding **/node_modules
Auto-excluding **/__pycache__
Auto-excluding **/.*
stubPath E:\Users\Avasam\Documents\Git\typeshed\typings is not a valid directory.
Searching for source files
Found 2 source files
pyright 1.1.285
E:\Users\Avasam\Documents\Git\typeshed\stubs\SQLAlchemy\@tests\test_cases\check_register.py
  E:\Users\Avasam\Documents\Git\typeshed\stubs\SQLAlchemy\@tests\test_cases\check_register.py:45:21 - error: Unnecessary "# type: ignore" comment

(my editor doesn't show the error, but that's a pylance limitation where it can only pickup the basic config file microsoft/pylance-release#3426 . rather look at the pyright run above)

My suspicions are confirmed — pyright should be emitting an error on the first commit of this PR about the unused type: ignore comment, but isn't:

• [DO-NOT-MERGE] Checking to see if pyright verifies third-party test cases against false negatives correctly #9623

python .\tests\pyright_test.py .\stubs\SQLAlchemy@tests -p .\pyrightconfig.testcases.json

Remember that we don't use this script in CI; we use pyright-action, which works differently

@AlexWaygood "enableTypeIgnoreComments": true specifically for test_cases. So it doesn't ignore it, with a comment to warn anyone who'd want to deactivate it:

As I explained above, the pyrightconfig.testcases.json file is only for the stdlib test cases. It isn't used for the third-party test cases, which was the whole reason why I added this check to check_consistent.py in the first place :)

Okay, so we can see in #9623 that pyright isn't verifying our third-party test cases guarding against false negatives in our CI. I also can't work out how to fix that. Adding # pyright: enableTypeIgnoreComments=true to the top of the file didn't make pyright pick up on the unused type: ignore comment. Neither have my various attempts to use glob patterns in pyrightconfig.testcases.json, to try to get pyright to use that config file for testing our third-party test cases. (The pyright docs say it should be possible; I don't know why it isn't working here.)

The good news is that pyright is verifying our stdlib test cases guarding against false negatives: see #9624.

Thank you for raising this issue! I'm closing this in favour of #9650, but I've listed you as a co-author over there :)