py/mkrules.mk: Use partial clone for submodules if available. #16163
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Code size report: |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #16163 +/- ##
=======================================
Coverage 98.57% 98.57%
=======================================
Files 164 164
Lines 21349 21349
=======================================
Hits 21045 21045
Misses 304 304 ☔ View full report in Codecov by Sentry. |
457663f to
c3571dd
Compare
projectgus
approved these changes
Dec 10, 2024
MicroPython relies on a number of submodules for third party and chip vendor libraries. Users need to check these out before building their desired ports and Github Actions CI here needs to clone them all multiple times for every build. Many of these are getting significantly larger over time, slowing down usage and consuming more disk space. Newer versions of git have features to avoid pulling all historic / blob data which can have a significant impact of total data use. This commit uses a standard feature of git to do a partial clone, with automatic fallback to previous behavior on error. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
c3571dd to
fdd606d
Compare
|
@andrewleech running Eg in |
|
Yeah I've seen that pop up in a couple of issue reports too! It certainly was unintentional, I didn't realise a comment there would be echo'd. I've seen comments above make targets before and thought that looked a bit ugly / out of context but now I understand why. |
|
I made a PR to fix this comment in #16595. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Micropython relies on a number of submodules for third party and chip vendor libraries. Users need to check these out before building their desired ports and Github Actions CI here needs to clone them all multiple times for every build.
Many of these are getting significantly larger over time, slowing down usage and consuming more disk space. There can also be resistance to adding larger libraries / sdk's here as submodules to avoid these sort of issues.
As suggested in #11349 (comment) by @dhalbert newer version of git have features to avoid pulling all historic / blob data which can have a significant impact of total data use.
The change here has been directly inspired by adafruit#7225 and adafruit#7788
Testing
Very little so far, it's been pushed here to trigger CI to ensure it still passes and hopefully get an idea of any speed improvements gained.
This is using a standard feature of git with automatic fallback to previous behavior on error.
Trade-offs and Alternatives
Shallow clone is the more common way of achieving this but that often results in issues due to the lack of history metadata, particularly around versions derived from tags etc. While that's less often an issue in submodules this filter method is generally considered a safer means of gaining similar size improvements.