Skip to content

ci: improve Go caching #7954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Jun 12, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 45 additions & 16 deletions .github/actions/setup-go/action.yaml
Original file line number Diff line number Diff line change
@@ -1,36 +1,65 @@
name: "Setup Go"
description: |
Sets up the Go environment for tests, builds, etc.
inputs:
version:
description: "The Go version to use."
default: "1.20.5"
runs:
using: "composite"
steps:
- name: Cache go toolchain
uses: buildjet/cache@v3
with:
path: |
${{ runner.tool_cache }}/go/${{ inputs.version }}
key: gotoolchain-${{ runner.os }}-${{ inputs.version }}
restore-keys: |
gotoolchain-${{ runner.os }}-

- uses: buildjet/setup-go@v4
with:
cache: true
go-version: "1.20.5"
# We do our own caching for implementation clarity.
cache: false
go-version: ${{ inputs.version }}

- name: Get cache dirs
shell: bash
run: |
set -x
echo "GOMODCACHE=$(go env GOMODCACHE)" >> $GITHUB_ENV
echo "GOCACHE=$(go env GOCACHE)" >> $GITHUB_ENV
Comment on lines +26 to +31
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work on Windows runners

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh?


# We split up GOMODCACHE from GOCACHE because the latter must be invalidated
# on code change, but the former can be kept.
- name: Cache $GOMODCACHE
uses: buildjet/cache@v3
with:
path: |
${{ env.GOMODCACHE }}
key: gomodcache-${{ runner.os }}-${{ hashFiles('**/go.sum') }}-${{ github.job }}
restore-keys: |
gomodcache-${{ runner.os }}-${{ hashFiles('**/go.sum') }}-
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GOCACHE also has the github.job one listed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a comment that may help this make more sense

gomodcache-${{ runner.os }}-

- name: Cache go
- name: Cache $GOCACHE
uses: buildjet/cache@v3
with:
# ~/go/pkg is the same across operating systems.
path: |
~/go/pkg
~/.cache/go-build
~/AppData/Local/go-build
~/Library/Caches/go-build
${{ env.GOCACHE }}
# Job name must be included in the key for effective
# test cache reuse.
key: go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/*.go', 'go.**') }}
# The key format is intentionally different than GOMODCACHE, because any
# time a Go file changes we invalidate this cache, whereas GOMODCACHE
# is only invalidated when go.sum changes.
key: gocache-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/*.go', 'go.**') }}
restore-keys: |
go-${{ runner.os }}-${{ github.job }}-
go-${{ runner.os }}-
go-
gocache-${{ runner.os }}-${{ github.job }}-
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hyphen at the end?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was on purpose. It's how I've seen it done on a few docs. I guess it clearly symbolizes the end of a token.

gocache-${{ runner.os }}-

- name: Install gotestsum
uses: jaxxstorm/action-install-gh-release@v1.10.0
with:
repo: gotestyourself/gotestsum
tag: v1.9.0
shell: bash
run: go install gotest.tools/gotestsum@latest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not work on Windows AFAIK

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?


# It isn't necessary that we ever do this, but it helps
# separate the "setup" from the "run" times.
Expand Down
86 changes: 34 additions & 52 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -100,49 +100,45 @@ jobs:

- uses: ./.github/actions/setup-go

# Check for any typos!
- uses: ./.github/actions/setup-node

- name: Get golangci-lint cache dir
run: |
go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.53.2
dir=$(golangci-lint cache status | awk '/Dir/ { print $2 }')
echo "LINT_CACHE_DIR=$dir" >> $GITHUB_ENV

- name: golangci-lint cache
uses: buildjet/cache@v3
with:
path: |
${{ env.LINT_CACHE_DIR }}
key: golangci-lint-${{ runner.os }}-${{ hashFiles('**/*.go') }}
restore-keys: |
golangci-lint-${{ runner.os }}-

# Check for any typos
- name: Check for typos
uses: crate-ci/typos@v1.14.12
with:
config: .github/workflows/typos.toml

- name: Fix the typos
if: ${{ failure() }}
run: |
echo "::notice:: you can automatically fix typos from your CLI:
cargo install typos-cli
typos -c .github/workflows/typos.toml -w"

# Check for Go linting errors!
- name: Lint Go
uses: golangci/golangci-lint-action@v3.5.0
with:
version: v1.52.2

- name: Lint shell scripts
uses: ludeeus/action-shellcheck@2.0.0
env:
SHELLCHECK_OPTS: --external-sources
with:
ignore: node_modules

- uses: ./.github/actions/setup-node
- name: Lint TypeScript
run: yarn lint
working-directory: site

# Make sure the Helm chart is linted!
# Needed for helm chart linting
- name: Install helm
uses: azure/setup-helm@v3
with:
version: v3.9.2
- name: Lint Helm chart
run: |
cd helm
make lint

# Ensure AGPL and Enterprise are separated!
- name: Check for AGPL code importing Enterprise...
run: ./scripts/check_enterprise_imports.sh
- name: make lint
run: |
make --output-sync=line -j lint

gen:
timeout-minutes: 8
Expand All @@ -158,16 +154,14 @@ jobs:
- name: Install sqlc
run: |
curl -sSL https://github.com/kyleconroy/sqlc/releases/download/v1.17.2/sqlc_1.17.2_linux_amd64.tar.gz | sudo tar -C /usr/bin -xz sqlc
- name: Install protoc-gen-go
run: go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.30
- name: Install protoc-gen-go-drpc
run: go install storj.io/drpc/cmd/protoc-gen-go-drpc@v0.0.33
- name: Install goimports
run: go install golang.org/x/tools/cmd/goimports@latest
- name: Install yq
run: go run github.com/mikefarah/yq/v4@v4.30.6
- name: Install mockgen
run: go install github.com/golang/mock/mockgen@v1.6.0

- name: go install tools
run: |
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.30
go install storj.io/drpc/cmd/protoc-gen-go-drpc@v0.0.33
go install golang.org/x/tools/cmd/goimports@latest
go install github.com/mikefarah/yq/v4@v4.30.6
go install github.com/golang/mock/mockgen@v1.6.0

- name: Install Protoc
run: |
Expand All @@ -189,7 +183,7 @@ jobs:
run: ./scripts/check_unstaged.sh

fmt:
runs-on: ubuntu-latest
runs-on: ${{ github.repository_owner == 'coder' && 'buildjet-8vcpu-ubuntu-2204' || 'ubuntu-latest' }}
timeout-minutes: 5
steps:
- name: Checkout
Expand Down Expand Up @@ -268,7 +262,7 @@ jobs:
go run ./scripts/ci-report/main.go gotests.json | tee gotests_stats.json

- uses: ./.github/actions/upload-datadog
if: always()
if: success() || failure()
with:
api-key: ${{ secrets.DATADOG_API_KEY }}

Expand Down Expand Up @@ -315,15 +309,8 @@ jobs:
# so we need to print the test stats to the log.
go run ./scripts/ci-report/main.go gotests.json | tee gotests_stats.json

- uses: actions/upload-artifact@v3
if: success() || failure()
with:
name: gotests-postgres.xml
path: ./gotests.xml
retention-days: 30

- uses: ./.github/actions/upload-datadog
if: always()
if: success() || failure()
with:
api-key: ${{ secrets.DATADOG_API_KEY }}

Expand All @@ -349,11 +336,6 @@ jobs:

- uses: ./.github/actions/setup-go

- uses: hashicorp/setup-terraform@v2
with:
terraform_version: 1.1.9
terraform_wrapper: false

- name: Run Tests
run: |
gotestsum --junitfile="gotests.xml" -- -race ./...
Expand Down
3 changes: 1 addition & 2 deletions .golangci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -200,12 +200,11 @@ issues:
max-same-issues: 0

run:
concurrency: 4
skip-dirs:
- node_modules
skip-files:
- scripts/rules.go
timeout: 5m
timeout: 10m

# Over time, add more and more linters from
# https://golangci-lint.run/usage/linters/ as the code improves.
Expand Down
13 changes: 12 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -402,11 +402,17 @@ else
endif
.PHONY: fmt/shfmt

lint: lint/shellcheck lint/go
lint: lint/shellcheck lint/go lint/ts lint/helm
.PHONY: lint

lint/ts:
cd site
yarn && yarn lint
.PHONY: lint/ts

lint/go:
./scripts/check_enterprise_imports.sh
go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.53.2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be skipped if golangci-lint is in PATH?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do it though? It runs almost instantly if already installed.

golangci-lint run
.PHONY: lint/go

Expand All @@ -416,6 +422,11 @@ lint/shellcheck: $(SHELL_SRC_FILES)
shellcheck --external-sources $(SHELL_SRC_FILES)
.PHONY: lint/shellcheck

lint/helm:
cd helm
make lint
.PHONY: lint/helm

# all gen targets should be added here and to gen/mark-fresh
gen: \
coderd/database/dump.sql \
Expand Down
3 changes: 1 addition & 2 deletions enterprise/cli/root_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,10 @@ package cli
import (
"testing"

"github.com/coder/coder/cli"
"github.com/stretchr/testify/require"

"github.com/coder/coder/cli"
"github.com/coder/coder/cli/clibase"

"github.com/coder/coder/cli/clitest"
)

Expand Down