Maybe it would be nice if we could have the CI complain if the % drops as a consequence of the changes? Although I'm not sure what the best way to implement something like that 🤔.

Maybe we already use basedpyright for this? It has some additional inference and narrowing features that might be helpful here, e.g. https://docs.basedpyright.com/latest/benefits-over-pyright/improved-generic-narrowing/. And if I understand correctly, then its baseline feature could be used as an alternative to --exclude-like.

sure but wouldn't we need DetachHead/basedpyright#125 to be closed first?

Maybe we already use basedpyright for this? It has some additional inference and narrowing features that might be helpful here, e.g. docs.basedpyright.com/latest/benefits-over-pyright/improved-generic-narrowing. And if I understand correctly, then its baseline feature could be used as an alternative to --exclude-like.

sure but wouldn't we need DetachHead/basedpyright#125 to be closed first?

No, --verifytypes already works. The idea behind DetachHead/basedpyright#125 is to make it possible for --verifytypes to be enabled as a diagnostic rule, so we can use e.g. # pyright: ignore[reportVerifyTypes] or pyright: reportVerifyTypes=false to ignore what we need to ignore. That way, we could configure it to only enable the reportVerifyTypes rule, and we just basedpyright numpy (without --verifytypes), which will then fail if any new errors pop up. We could use the baseline feature to generate all of the initial ignore's, so no need for manual labor.

thanks - i've tried running

spin run python -m  basedpyright  --verifytypes numpy --writebaseline

but I'm not getting any outputs written anywhere

the docs mention ./.basedpyright/baseline.json, but I don't see it

(numpy-dev) marcogorelli@DESKTOP-U8OKFP3:~/numpy-dev$ ls ./.basedpyright
ls: cannot access './.basedpyright': No such file or directory

can you see what i might be doing wrong?

sorry for the delay, had pycon italy + some PTO

Hmm yea --writebaseline doesn't seem to work when combined with --verifytypes.
Is there something we're doing wrong here, @DetachHead ?

without --verifytypes it seems to work, and it produces a .basedpyright/baseline.json of 30.4 MB (!), for the 11,719 errors and 117,354 warnings. But that also includes all .py sources and typing tests, so I added this to the pyproject.toml

[tool.basedpyright]
include = ["numpy"]
ignore = ["**/*.py", "**/tests/*"]
exclude = [".spin", "benchmarks", "build", "build-install", "doc", "tools"]
stubPath = "."
pythonPlatform = "All"

typeCheckingMode = "strict"
disableBytesTypePromotions = true
enableReachabilityAnalysis = false
enableTypeIgnoreComments = false
reportIgnoreCommentWithoutRule = true
reportPropertyTypeMismatch = false
reportPrivateUsage = false
reportShadowedImports = false
reportUnnecessaryTypeIgnoreComment = true

and ran it again with basedpyright --writebaseline to get a ~400 kB baseline.json of 1663 errors.

However, if I now run basedpyright --verifytypes numpy (which requires numpy to be installed, but not in editable mode), I see the exact same 22.6% "type completeness score" as I saw without the baseline.json. So I suppose that'll have to wait until DetachHead/basedpyright#125 then.

sure - i'll close this then in favour of a basedpyright solution when it's available

--verifytypes is a completely different thing to the normal type checker for some reason, so it probably doesn't work with baseline unfortunately