I am interested in looking into this, as I was a bit surprised that the testpep561.py file parsed the pep561.test file by direct reference (as do other tests I presume?). As I've read they do this by specifying the files property in a DataSuite-inheriting class. I've worked with code running in this extract (testpep561.py):

class PEP561Suite(DataSuite):
    files = ['pep561.test']

    def run_case(self, test_case: DataDrivenTestCase) -> None:
        test_pep561(test_case)

One would expect one (or a couple extended) classes(s) to parse all the tests and generate a recognizable pytest output.

It's going to require a bit more understanding on my side, although I gained some experience through #8524 and @ethanhs his work in #8571. So just for clarity's sake:

• We are interested in intercepting pytest autodiscovery for data-driven tests (any class extending Suite/DataSuite/etc.) . Any pointers to which class to start studying if so? (File: data.py or helpers.py possibly).
• We are interested in combining unit tests starting with foo by regex'ing foo-*.test files.
• We need some scanning or mapping state of which tests are picked up only once by a module (f.e. mypy.test.testcheck as your example goes). I'd expect this to be more strict: we must enforce a .test file to map to one test*.py file only. Only then can we make sure the data-driven tests are run by at most one module. A second requirement of course is that the unit test maps to the right test module. A third is that if the test-data can regex to multiple modules, an error must be thrown. Does this sound complete/correct?
• We'd like some way of generating a mapping of the data unit tests in a custom discovery class, with 'missed_cases' and 'multiple_mappings' as output, but it is expected to by tricky because it can generate wrong information/advice or has some edge cases.

@JukkaL it'd be awesome to hear your thoughts so I can get to work.

Just a 'go-ahead' is fine for now.

I will work on this, although any comment would be great. @ethanhs do you have any thoughts? It isn't a small endeavor this time, but pretty similar to data-driven tests like we worked on earlier

I'm afraid I'm not sure the particulars of what @JukkaL has in mind, so it'd be best to ask him.

@JukkaL I'd gladly hear about whether this issue can be acted upon

I'd like to work on this issue.
From what I understand, we need the list of .test files to be auto generated. I am thinking of using something like pathlib.Path().rglob("prefix*.test") to do so. What do you think?

Up for grabs, @ethanhs @JukkaL could you help this fella out?

Hello Sir ,

I was trying to fix auto-discovery of .test files #8650

Here is the script

# List of files that contain test case descriptions.

typecheck_files = [f for f in os.listdir("../../test-data/unit") if re.match(r'check-\w+-?\w+?.test',f)]

# Tests that use Python 3.8-only AST features (like expression-scoped ignores):
if sys.version_info < (3, 8):
    typecheck_files.remove('check-python38.test')
if sys.version_info < (3, 9):
    typecheck_files.remove('check-python39.test')

# Special tests for platforms with case-insensitive filesystems.
if sys.platform not in ('darwin', 'win32'):
    typecheck_files.remove('check-modules-case.test')  

But there was path not found error. Can you exactly tell me from where would it be called ?
As seen in ReadMe , you have called many functions as

$ pytest -n0 mypy/test/testcheck.py

So is that the default location from where we can call it ? If yes ,then maybe this would work :

# List of files that contain test case descriptions.

typecheck_files = [f for f in os.listdir("test-data/unit") if re.match(r'check-\w+-?\w+?.test', f)]


# Tests that use Python 3.8-only AST features (like expression-scoped ignores):
if sys.version_info < (3, 8):
    typecheck_files.remove('check-python38.test')
if sys.version_info < (3, 9):
    typecheck_files.remove('check-python39.test')

# Special tests for platforms with case-insensitive filesystems.
if sys.platform not in ('darwin', 'win32'):
    typecheck_files.remove('check-modules-case.test')
    

Please Help !!

Hello, I'd like to solve this issue, although many have already solved this prior to me.
@JukkaL @davidzwa Can anyone reach out, I have a few pointers. Also, it'd be great to start my contribution journey from this issue of mypy.

Thanks!