python/typeshed#11830 (comment) and python/typeshed#13021 (comment) are examples of how it's used by typeshed

This would be very useful. It may also be helpful to do this on a cron against the nightly wheels. And we may need to choose mypy vs pyright, depending on what the project itself uses (they may no longer be making an effort to be compatible with mypy, e.g.).

It may also be helpful to do this on a cron against the nightly wheels.

There will be cases when we will deliberately choose to ignore the mypy_primer output. So I can imagine that doing this periodically would be more annoying than it would be helpful 🤔.

And we may need to choose mypy vs pyright, depending on what the project itself uses (they may no longer be making an effort to be compatible with mypy, e.g.).

Unlike the name suggests, mypy_primer isn't exclusive to mypy. If I understand correctly, for each project it can be configured to use mypy, pyright, both, or forks (e.g. basedmypy or basedpyright).

Isn't the problem that there are very few external projects to run CI on that actually have good type annotations? I'm thinking that a single job to sometimes run against scipy-stubs (anything else?) that you trust is good, but much more is going to be a pain.

The usual solution for this problem that we encourage is for downstream packages to test numpy nightlies. Quite a few projects do this now, so getting them to run their typing tests as well on those nightlies if they have any is probably more effective.

Yes, we'd be happy to add a daily/weekly check like that to skimage and the SP Lecture Notes, e.g.

Isn't the problem that there are very few external projects to run CI on that actually have good type annotations?

It doesn't matter if the projects are currently correctly typed, or even typed at all. Mypy_primer reports the diffs of the mypy/pyright outputs of that project, that our change causes.

Awesome that you already thought of this! I went ahead and added a few features to mypy_primer to make this easy to add to numpy.

• --known-dependency-selector numpy. This lets you select all projects in mypy_primer's corpus where we at some point recorded that numpy is a dependency. There are currently 19 of these.
• --(old/new)-prepend-path. This lets you effectively add a new entry to sys.path for the new and old runs. Two gotchas: a) you'll want to make sure there's a numpy dir and a numpy/py.typed in the path you prepend, b) there may be some weirdness with namespace packages (which I guess doesn't apply to numpy)

I was able to get some potentially believable output with:

mypy_primer --type-checker mypy --new v1.14.0 --old v1.14.0 --known-dependency-selector numpy --old-prepend-path ~/tmp/nump1/numpy --new-prepend-path ~/tmp/nump2/numpy --output concise

where I checked out numpy v2.1.3 and v2.2.0 to nump1 and nump2 respectively. Make sure to explicitly set --new and --old to the same thing, otherwise you'll also get type checker version differences.

mypy_primer supports pyright and mypy. It probably also supports basedpyright via the --repo flag.

I do see mypy hit some errors when using v2.1.3 on the lines of AssertionError: numpy._typing._nbit_base._8Bit. This is somewhat surprising, because we should have seen those in mypy and typeshed CI. Any chance that error is at all familiar to you? It's possible I messed something up.

Anything I should know before opening a PR?

Wow, thanks a lot for that @hauntsaninja, that indeed sounds very helpful.

I do see mypy hit some errors when using v2.1.3 on the lines of AssertionError: numpy._typing._nbit_base._8Bit. This is somewhat surprising, because we should have seen those in mypy and typeshed CI. Any chance that error is at all familiar to you? It's possible I messed something up.

Hmm, maybe it has something to do with the mypy plugin (https://github.com/numpy/numpy/blob/main/numpy/typing/mypy_plugin.py)? It is used for system-dependent scalar types, like np.longdouble, so that the types match match the runtime, for which it uses those _typing._nbit_base types. But I don't see any asserts in the mypy plugin itself, so the AssertionError could then perhaps come from mypy itself or something?
🤷🏻

Anything I should know before opening a PR?

Nothing in particular I suppose.
Just don't forget to use a CI: prefix for your commit messages and PR title (or whichever one seems appropriate, see https://numpy.org/doc/stable/dev/development_workflow.html#writing-the-commit-message).

Thanks! Ah, the plugin is a good guess. It looks like the plugin imports numpy but I didn't bother rebuilding numpy, so it's possible I introduced some skew.