Skip to content

Prune non-wildcard excludePaths directories early #2503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from

Conversation

mvorisek
Copy link
Contributor

@mvorisek mvorisek commented Jul 1, 2023

uses backported symfony/symfony/pull/50877 with symfony/symfony/pull/50884 optimization

fix phpstan/phpstan#9554

in projects with massive amount of excluded files, this PR improves phpstan runtime by several times - in our case, the phpstan file scan speedup is 280 times (from 48 seconds to 0.17 seconds)

fully tested using VFS

@mvorisek mvorisek changed the base branch from 1.11.x to 1.10.x July 1, 2023 20:57
@phpstan-bot
Copy link
Collaborator

You've opened the pull request against the latest branch 1.11.x. If your code is relevant on 1.10.x and you want it to be released sooner, please rebase your pull request and change its target to 1.10.x.

@mvorisek mvorisek force-pushed the prune_early branch 3 times, most recently from 267329d to 3e67bbf Compare July 1, 2023 21:21
@mvorisek mvorisek changed the title Prune non-wildcard directory excludePaths early Prune non-wildcard excludePaths directories early Jul 1, 2023
@mvorisek mvorisek marked this pull request as ready for review July 1, 2023 21:23
@phpstan-bot
Copy link
Collaborator

This pull request has been marked as ready for review.

@mvorisek mvorisek force-pushed the prune_early branch 2 times, most recently from 6e55b7a to 57eb0e4 Compare July 1, 2023 21:38
Copy link
Member

@ondrejmirtes ondrejmirtes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes need a test to show why they're done.

It can be a test calling PHPStan from the outside, there's plenty of those in phpstan/phpstan.

@mvorisek mvorisek marked this pull request as draft July 2, 2023 16:20
@mvorisek mvorisek force-pushed the prune_early branch 8 times, most recently from da56f5b to 1feecdd Compare July 4, 2023 10:31
@mvorisek mvorisek marked this pull request as ready for review July 4, 2023 10:35
@phpstan-bot
Copy link
Collaborator

This pull request has been marked as ready for review.

@mvorisek mvorisek marked this pull request as draft July 6, 2023 16:46
@mvorisek mvorisek force-pushed the prune_early branch 6 times, most recently from 12a7567 to 82660e7 Compare July 6, 2023 18:49
@mvorisek mvorisek marked this pull request as ready for review July 6, 2023 18:55
@phpstan-bot
Copy link
Collaborator

This pull request has been marked as ready for review.

@mvorisek mvorisek requested a review from ondrejmirtes July 6, 2023 18:57
@mvorisek
Copy link
Contributor Author

mvorisek commented Jul 6, 2023

These changes need a test to show why they're done.

done, tested using VFS, so minimal count of IO syscalls is asserted :)

@mvorisek
Copy link
Contributor Author

@ondrejmirtes this PR solves a major performance issue and in our case, it cuts the phpstan runtime by more than 20x, would you please mind reviwing this PR soon? Thank you.

@ondrejmirtes
Copy link
Member

Would be great if you were able to backport symfony/symfony#50877 into Symfony 5.4, then we could just use their code without all the complex VFS tests.

@mvorisek
Copy link
Contributor Author

mvorisek commented Nov 3, 2023

From Symfony perspective it is a feature and 5.x branch does no longer accept features. Is copying the pruning filtering here ok or when https://github.com/symfony/symfony/releases/tag/v6.4.0-BETA2 (or Symfony 6.4 in general) can be required in phpstan?

@ondrejmirtes
Copy link
Member

Not anytime soon, PHPStan still supports PHP 7.2+.

@mvorisek
Copy link
Contributor Author

This PR backports the needed class incl. tests. Can we merge it?

In our case, the file discovery speedup is 280 times.

Once Symfony 6.4+ could be used, we can remove the backported class.

@ondrejmirtes
Copy link
Member

speedup is 280 times.

What it means in absolute numbers? 1ms vs. 280ms I'm not interested in but if the numbers are higher then for sure.

@mvorisek
Copy link
Contributor Author

see PR description - in our case, from 48 seconds to 0.17 seconds, the actual numbers depend on the files ignored

@ondrejmirtes
Copy link
Member

Sure we could merge this, but at the same time you could change your PHPStan config so that you don't have "massive amount of excluded files". You could even change your project structure so that all your own is under src/ or app/ as you typically do with PSR-4.

@mvorisek
Copy link
Contributor Author

mvorisek commented Dec 6, 2023

Let's merge this PR, it is highly supported by the community.

@ondrejmirtes
Copy link
Member

No, sorry, I can't merge it in the current state. It's too risky, I'm afrait of too much different behaviour of the newly introduced FileFinderSymfonyBackport class. It's 170 lines of code that looks for files completely differently than the currently used Symfony Finder class (which is 800+ different lines of code).

Compare that to your Symfony PR - https://github.com/symfony/symfony/pull/50877/files - it's just a 40 lines of actual production code changes in src/Symfony/Component/Finder/Finder.php and src/Symfony/Component/Finder/Iterator/ExcludeDirectoryFilterIterator.php.

If you'd write a Symfony\Component\Finder\Finder subclass that would only override the methods that it needs to override in order to achieve early pruning, it'd be much easier for me to merge this.

@mvorisek
Copy link
Contributor Author

mvorisek commented Dec 6, 2023

Except fa5ded5 which can be reverted/dropped, it cannot be simplifed more. It is trully a backport of the newer Symfony. If you want this change, please either merge it as is or backport from scratch. If you do not want the change, close this PR. I will drop a tear but it is your repo.

@ondrejmirtes
Copy link
Member

This is not a backport. A backport would if you copied or extended this class https://github.com/symfony/symfony/blob/6.4/src/Symfony/Component/Finder/Finder.php and made it work with PHP 7.2+ and in context of PHPStan codebase as it is.

I can't risk merging this. When I get a hunch some PR would cause problems, I'm usually right. I can't afford breaking users' analysis and spending the next week fixing bugs in code I don't understand.

To be able to merge this PR I'd have to see that the finder finds all the same files and finds them in the same order across various projects with complex configurations. But I don't see that here, sorry.

@mvorisek
Copy link
Contributor Author

mvorisek commented Dec 7, 2023

This is not a backport. A backport would if you copied or extended this class https://github.com/symfony/symfony/blob/6.4/src/Symfony/Component/Finder/Finder.php and made it work with PHP 7.2+ and in context of PHPStan codebase as it is.

This is exactly what I did. I copied methods that are needed to backport the new feature, I extended the base class, as not all methods were needed to be copied. Also, with VFS test, I have prooven that the behaviour is the same as the one in Symfony, same test I contributed there.

@ondrejmirtes
Copy link
Member

I extended the base class

The finder class introduced in this PR does not extend any class: class FileFinderSymfonyBackport implements IteratorAggregate

@mvorisek
Copy link
Contributor Author

mvorisek commented Dec 7, 2023

The reason is https://github.com/symfony/symfony/blob/v7.0.1/src/Symfony/Component/Finder/Finder.php#L745 is private and implementing all Finder functionality will result in more LOC copied and untested. Thus I copied only the relevant/needed methods. Help with backporting symfony/symfony#50877 welcomed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Files discovery must prune early
3 participants