Currently, the auto-discovery run is not cancellable. This can some times get in the way when say pytest is not installed , and you install pytest it can prevent from running immediately. May be, we can check if pytest exists or if path is provided, if eithe is true then run discovery using withProgress API so even the auto discover can be cancelled.