All commit authors signed the Contributor License Agreement.

I just signed https://www.python.org/psf/contrib/contrib-form/, I hope that will fix CI eventually.

Thanks for adding this! I looked over the test file changes and they mostly look good, as noted above. I didn't review all the results for Zuban.

One concern: I know you're using a partly paid model, where some users have to pay for usage of the type checker. How does that system interact with usage of Zuban in this repo? For example, could we run into issues if the conformance suite grows past some line count, or if we tried to run all type checkers in CI?

Thanks for adding this.

I didn't do a line-by-line review, but I did skim through it. I have a few questions:

1. I noticed that many of the "Notes" fields in the Zuban results are identical to the "Notes" field for mypy. Is this correct? Unless Zuban is based on mypy directly (and is therefore bug-for-bug compatible), it seems unlikely that the conformance failure modes would be identical between the two type checkers.
2. I noticed that some of the tests that were scored as "Partial" are missing a "Notes" field. For consistency, please provide a "Notes" field for all test cases that are not scored as "Pass". These notes appear in the conformance summary report when you mouse over a box that says "Partial".

One concern: I know you're using a partly paid model, where some users have to pay for usage of the type checker. How does that system interact with usage of Zuban in this repo? For example, could we run into issues if the conformance suite grows past some line count, or if we tried to run all type checkers in CI?

No, I'm most likely going to open source it anyway in the near future and at the moment there the line count limits are disabled.

I noticed that many of the "Notes" fields in the Zuban results are identical to the "Notes" field for mypy. Is this correct? Unless Zuban is based on mypy directly (and is therefore bug-for-bug compatible), it seems unlikely that the conformance failure modes would be identical between the two type checkers.

It is based on Mypy's test suite, so if you look at the automatic test results and open them side by side you quite often see the exact same results. If the results are not the same the issues are sometimes still the same even if there is an additional error, it still stems from the same fact (e.g. some sort of thing is not supported, because it's not covered by Mypy's test suite).

I also specifically fixed the issues that would differ from Mypy's results to be more similar to Mypy, so I wouldn't have to write my own Notes. I'm lazy in a weird way :)

I noticed that some of the tests that were scored as "Partial" are missing a "Notes" field. For consistency, please provide a "Notes" field for all test cases that are not scored as "Pass". These notes appear in the conformance summary report when you mouse over a box that says "Partial".

I will add notes.

I'm concerned about adding support for a type checker with a proprietary license. I'm volunteering my (and my company's) time for work on typing, and the contract has always been that the software we're directly supporting is licensed freely. For example, while we're directly supporting pyright, which is MIT licensed, we're not directly supporting the proprietary pylance language server. (Of course, we're indirectly supporting proprietary software and improving typing as a whole and fixing bugs reported by developers and users of proprietary software, but that benefits all typing users.)

Frankly speaking, if I'm supposed to support proprietary software, me or my company expects to be paid for it. So, I'm -1 on merging this (and any similar) PRs.

@erictraut @JelleZijlstra I think I fixed the remaining issues.

@srittau Did you read the part about open sourcing?

If Zuban would go Open Source, my concerns would obviously be irrelevant.

Does anybody know why the CLA part of CI is still failing? I signed it yesterday.

I hoped that closing and reopening would fix the CLA issue but no dice. @ambv could you help take a look?

No, I'm most likely going to open source it anyway in the near future and at the moment there the line count limits are disabled.

That's good to know!

@srittau I broadly agree with this concern (and I probably wouldn't support adding Zuban to typeshed CI while it's still proprietary), but I think it's not too bad for the conformance suite, where we don't have to do that much more than run the type checker and record the results.

I just signed https://www.python.org/psf/contrib/contrib-form/, I hope that will fix CI eventually.

It looks like the CLA form you signed requires manual processing, which probably hasn't gone through yet. The other way to sign the CLA is by clicking the button in the bot comment / visiting https://cla.python.org/, which is a fully automated process

@srittau I broadly agree with this concern (and I probably wouldn't support adding Zuban to typeshed CI while it's still proprietary), but I think it's not too bad for the conformance suite, where we don't have to do that much more than run the type checker and record the results.

Fair enough, especially since I personally stay away from the conformance tests.

@brianschubert Thanks that went through.
@srittau @JelleZijlstra I wouldn't even have tried to add Zuban to typeshed, because I know how much work that would be for you guys and appreciate what you are working on. I think here it's a bit different, but I doubt that matters in a few weeks.