Skip to content

chore: persist go build cache between CI runs on Windows #17614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

hugodutka
Copy link
Contributor

No description provided.

@hugodutka hugodutka force-pushed the hugodutka/persist-go-build-cache branch 6 times, most recently from 2ed6ff3 to 8608245 Compare April 30, 2025 12:54
@@ -313,7 +313,7 @@ jobs:
run: ./scripts/check_unstaged.sh

test-go:
runs-on: ${{ matrix.os == 'ubuntu-latest' && github.repository_owner == 'coder' && 'depot-ubuntu-22.04-4' || matrix.os == 'macos-latest' && github.repository_owner == 'coder' && 'depot-macos-latest' || matrix.os == 'windows-2022' && github.repository_owner == 'coder' && 'windows-latest-16-cores' || matrix.os }}
runs-on: ${{ matrix.os == 'ubuntu-latest' && github.repository_owner == 'coder' && 'depot-ubuntu-22.04-4' || matrix.os == 'macos-latest' && github.repository_owner == 'coder' && 'depot-macos-latest' || matrix.os == 'windows-2022' && github.repository_owner == 'coder' && 'depot-windows-2022-16' || matrix.os }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ thank you! I was planning on doing this, too.

@hugodutka hugodutka force-pushed the hugodutka/persist-go-build-cache branch 18 times, most recently from 7781388 to 2affa44 Compare April 30, 2025 20:19
@hugodutka hugodutka force-pushed the hugodutka/persist-go-build-cache branch from 2affa44 to 9c808b4 Compare April 30, 2025 20:44
@hugodutka hugodutka force-pushed the hugodutka/persist-go-build-cache branch from bd9677c to 6db82ca Compare May 1, 2025 10:13
@hugodutka hugodutka force-pushed the hugodutka/persist-go-build-cache branch 3 times, most recently from 5b863af to e70d793 Compare May 1, 2025 11:07
@hugodutka
Copy link
Contributor Author

I'm trying to enable test caching between CI runs on Windows as discussed on Slack. It turned out to be a bit tricky, and it's still a work in progress. Here are some of the problems I ran into:

  1. Even though the setup-go action caches the GOCACHE directory, it only does so when go.sum files change. We need to cache test results more frequently.

  2. To address point 1, I leveraged actions/cache directly. However, the GOCACHE directory is about 1.5GB in size. Because Windows disks on GitHub action runners (Depot included) are notoriously slow, it takes a minute and a half to restore it. I think most of the time is spent on unarchiving.

  3. A Go test is cached if none of its inputs change. A test's inputs include any environment variables it reads, files it reads from disk (if they are within the test's package directory), and code dependencies.

    • Many of our tests read the PATH variable. However, on Windows, the setup-terraform action updates the PATH variable with a directory that's a randomly generated UUID—the directory that contains the terraform binary. This means any test that reads the PATH variable will not be cached.
    • The go test runtime determines whether a file input has changed based not on the file's contents, but on its last modified timestamp (and a couple of other attributes). Git does not store these timestamps, so they are all set to the current date and time. In order for tests to be cached, these timestamps have to be normalized.

I've developed PoC solutions to address these issues, but I still have some work to do to make them ready to merge.

@hugodutka hugodutka force-pushed the hugodutka/persist-go-build-cache branch from 7fe08b1 to 721d476 Compare May 1, 2025 18:33
@hugodutka hugodutka force-pushed the hugodutka/persist-go-build-cache branch 5 times, most recently from b26b034 to f347a53 Compare May 1, 2025 20:10
@hugodutka hugodutka force-pushed the hugodutka/persist-go-build-cache branch from f347a53 to 2e0f7b3 Compare May 1, 2025 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants