Skip to content

chore: persist go build cache between CI runs on Windows #17614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

chore: persist go build cache between CI runs on Windows #17614

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9c808b4
persist go build cache
hugodutka Apr 30, 2025
6db82ca
terraform
hugodutka May 1, 2025
79f1e47
ramdisk
hugodutka May 1, 2025
198112f
wush
hugodutka May 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
wush
  • Loading branch information
@hugodutka
hugodutka committed May 3, 2025
commit 198112f3ab0550b88112f16b7e020cbc4607fd15
46 changes: 23 additions & 23 deletions .github/actions/setup-go/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,27 +20,27 @@ runs:
with:
go-version: ${{ inputs.version }}
cache: ${{ inputs.cache }}
- name: Install gotestsum and mtimehash
shell: bash
run: |
export BUILD_CACHE_PATH=${{ inputs.build-cache-path }}
if [ -n "${BUILD_CACHE_PATH}" ]; then
mkdir -p "${BUILD_CACHE_PATH}"
export GOCACHE="${BUILD_CACHE_PATH}/build"
export GOMODCACHE="${BUILD_CACHE_PATH}/mod"
fi
go install gotest.tools/gotestsum@latest
go install github.com/slsyy/mtimehash/cmd/mtimehash@latest
# - name: Install gotestsum and mtimehash
# shell: bash
# run: |
# export BUILD_CACHE_PATH=${{ inputs.build-cache-path }}
# if [ -n "${BUILD_CACHE_PATH}" ]; then
# mkdir -p "${BUILD_CACHE_PATH}"
# export GOCACHE="${BUILD_CACHE_PATH}/build"
# export GOMODCACHE="${BUILD_CACHE_PATH}/mod"
# fi
# go install gotest.tools/gotestsum@latest
# go install github.com/slsyy/mtimehash/cmd/mtimehash@latest

# It isn't necessary that we ever do this, but it helps
# separate the "setup" from the "run" times.
- name: go mod download
shell: bash
run: |
export BUILD_CACHE_PATH=${{ inputs.build-cache-path }}
if [ -n "${BUILD_CACHE_PATH}" ]; then
mkdir -p "${BUILD_CACHE_PATH}"
export GOCACHE="${BUILD_CACHE_PATH}/build"
export GOMODCACHE="${BUILD_CACHE_PATH}/mod"
fi
go mod download -x
# # It isn't necessary that we ever do this, but it helps
# # separate the "setup" from the "run" times.
# - name: go mod download
# shell: bash
# run: |
# export BUILD_CACHE_PATH=${{ inputs.build-cache-path }}
# if [ -n "${BUILD_CACHE_PATH}" ]; then
# mkdir -p "${BUILD_CACHE_PATH}"
# export GOCACHE="${BUILD_CACHE_PATH}/build"
# export GOMODCACHE="${BUILD_CACHE_PATH}/mod"
# fi
# go mod download -x
25 changes: 23 additions & 2 deletions .github/actions/setup-imdisk/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: "Setup ImDisk"
if: runner.os == 'Windows'
description: |
Sets up the ImDisk toolkit for Windows and creates a RAM disk on drive R:.
Sets up the ImDisk toolkit for Windows and backs often-used paths with RAM disks
runs:
using: "composite"
steps:
Expand All @@ -22,6 +22,27 @@ runs:
install.bat /silent

- name: Create RAM Disk
shell: cmd
shell: bash
run: |
imdisk -a -s 4096M -m R: -p "/fs:ntfs /q /y"

# bash
mkdir "$RUNNER_TEMP"_tmp
cp -r "$RUNNER_TEMP"/. "$RUNNER_TEMP"_tmp
rm -rf "$RUNNER_TEMP"/*
imdisk -a -s 8192M -m "$RUNNER_TEMP" -p "/fs:ntfs /q /y"
cp -r "$RUNNER_TEMP"_tmp/. "$RUNNER_TEMP"
rm -rf "$RUNNER_TEMP"_tmp

cd "$RUNNER_TEMP"
export GOCACHE="$(go env GOCACHE)"
export GOMODCACHE="$(go env GOMODCACHE)"
rm -rf "$GOCACHE"
rm -rf "$GOMODCACHE"
mkdir -p "$GOCACHE"
mkdir -p "$GOMODCACHE"
imdisk -a -s 8192M -m "$GOCACHE" -p "/fs:ntfs /q /y"
imdisk -a -s 8192M -m "$GOMODCACHE" -p "/fs:ntfs /q /y"

rm -rf "$GITHUB_WORKSPACE"/*
imdisk -a -s 2048M -m "$GITHUB_WORKSPACE" -p "/fs:ntfs /q /y"
117 changes: 13 additions & 104 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,45 +236,6 @@ jobs:
with:
fetch-depth: 1

- name: Setup Node
uses: ./.github/actions/setup-node

- name: Setup Go
uses: ./.github/actions/setup-go

- name: Setup sqlc
uses: ./.github/actions/setup-sqlc

- name: Setup Terraform
uses: ./.github/actions/setup-tf

- name: go install tools
uses: ./.github/actions/setup-go-tools

- name: Install Protoc
run: |
mkdir -p /tmp/proto
pushd /tmp/proto
curl -L -o protoc.zip https://github.com/protocolbuffers/protobuf/releases/download/v23.4/protoc-23.4-linux-x86_64.zip
unzip protoc.zip
cp -r ./bin/* /usr/local/bin
cp -r ./include /usr/local/bin/include
popd

- name: make gen
run: |
# Remove golden files to detect discrepancy in generated files.
make clean/golden-files
# Notifications require DB, we could start a DB instance here but
# let's just restore for now.
git checkout -- coderd/notifications/testdata/rendered-templates
# no `-j` flag as `make` fails with:
# coderd/rbac/object_gen.go:1:1: syntax error: package statement must be first
make --output-sync -B gen

- name: Check for unstaged files
run: ./scripts/check_unstaged.sh

fmt:
needs: changes
if: needs.changes.outputs.offlinedocs-only == 'false' || needs.changes.outputs.ci == 'true' || github.ref == 'refs/heads/main'
Expand Down Expand Up @@ -316,7 +277,7 @@ jobs:
runs-on: ${{ matrix.os == 'ubuntu-latest' && github.repository_owner == 'coder' && 'depot-ubuntu-22.04-4' || matrix.os == 'macos-latest' && github.repository_owner == 'coder' && 'depot-macos-latest' || matrix.os == 'windows-2022' && github.repository_owner == 'coder' && 'depot-windows-2022-16' || matrix.os }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ thank you! I was planning on doing this, too.

needs: changes
if: needs.changes.outputs.go == 'true' || needs.changes.outputs.ci == 'true' || github.ref == 'refs/heads/main'
timeout-minutes: 20
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
Expand All @@ -326,88 +287,36 @@ jobs:
- windows-2022
steps:
- name: Harden Runner
if: runner.os == 'Linux'
uses: step-security/harden-runner@0634a2670c59f64b4a01f0f96f84700a4088b9f0 # v2.12.0
with:
egress-policy: audit

- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 1

# Sets up the ImDisk toolkit for Windows and creates a RAM disk on drive R:.
- name: Setup ImDisk
if: runner.os == 'Windows'
uses: ./.github/actions/setup-imdisk
uses: hugodutka/setup-imdisk@e6437cdfa0a7e110bc3f560da8c04b3c3fcc4135

- name: Download Go Build Cache
id: download-go-build-cache
uses: ./.github/actions/test-cache/download
if: runner.os == 'Windows'
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
key-prefix: test-go-build-11-${{ runner.os }}-${{ runner.arch }}
cache-path: "/r/.cache/go-cache"
fetch-depth: 1

- name: Setup Go
uses: ./.github/actions/setup-go
with:
build-cache-path: /r/.cache/go-cache
version: ""
cache: false

- name: Normalize File and Directory Timestamps
- name: Wush
if: always() && runner.os == 'Windows'
shell: bash
run: |
find . -type f ! -path ./.git/\*\* | mtimehash
find . -type d ! -path ./.git/\*\* -exec touch -t 200601010000 {} +

- name: Setup Terraform
uses: ./.github/actions/setup-tf

- name: Download Test Cache
id: download-cache
uses: ./.github/actions/test-cache/download
with:
key-prefix: test-go-${{ runner.os }}-${{ runner.arch }}

# - name: Test with Mock Database
# id: test
# shell: bash
# run: |
# # if macOS, install google-chrome for scaletests. As another concern,
# # should we really have this kind of external dependency requirement
# # on standard CI?
# if [ "${{ matrix.os }}" == "macos-latest" ]; then
# brew install google-chrome
# fi

# # By default Go will use the number of logical CPUs, which
# # is a fine default.
# PARALLEL_FLAG=""

# # macOS will output "The default interactive shell is now zsh"
# # intermittently in CI...
# if [ "${{ matrix.os }}" == "macos-latest" ]; then
# touch ~/.bash_profile && echo "export BASH_SILENCE_DEPRECATION_WARNING=1" >> ~/.bash_profile
# fi
# export TS_DEBUG_DISCO=true
# export GOCACHE=/r/.cache/go-cache/build
# export GOMODCACHE=/r/.cache/go-cache/mod
# echo "PATH: $PATH"
# gotestsum --junitfile="gotests.xml" --jsonfile="gotests.json" \
# --packages="./..." -- $PARALLEL_FLAG -short -failfast

- name: Upload Test Cache
uses: ./.github/actions/test-cache/upload
with:
cache-key: ${{ steps.download-cache.outputs.cache-key }}
curl -L -o wush.zip https://github.com/coder/wush/releases/download/v0.4.1/wush_0.4.1_windows_amd64.zip
unzip -o wush.zip

- name: Upload Go Build Cache
if: runner.os == 'Windows'
uses: ./.github/actions/test-cache/upload
with:
cache-key: ${{ steps.download-go-build-cache.outputs.cache-key }}-${{ github.run_id }}-${{ github.run_attempt }}
cache-path: "/r/.cache/go-cache"
override-condition: "true"
chmod +x wush.exe
./wush.exe serve

# We don't run the full test-suite for Windows & MacOS, so we just run the CLI tests on every PR.
# We run the test suite in test-go-pg, including CLI.
Expand Down
Loading