chore: run test-go-pg on macOS and Windows in regular CI #17853

hugodutka · 2025-05-15T13:13:30Z

This PR starts running test-go-pg on macOS and Windows in regular CI. Previously this suite was only run in the nightly gauntlet for 2 reasons:

it was flaky
it was slow (took 17 minutes)

We've since stabilized the flakiness by switching to depot runners, using ram disks, optimizing the number of tests run in parallel, and automatically re-running failing tests. We've also brought down the time to run the suite to 9 minutes. Additionally, this PR allows test-go-pg to use cache from previous runs, which speeds it up further. The cache is only used on PRs, main will still run tests without it.

This PR also:

removes the nightly gauntlet since all tests now run in regular CI
removes the test-cli job for the same reason
removes the setup-imdisk action which is now fully replaced by coder/setup-ramdisk-action
makes 2 minor changes which could be separate PRs, but I rolled them into this because they were helpful when iterating on it:
- replace the if: always() condition on the gen job with a if: ${{ !cancelled() }} to allow the job to be cancelled. Previously the job would run to completion even if the entire workflow was cancelled. See the GitHub docs for more details.
- disable the recently added TestReinitializeAgent since it does not pass on Windows with Postgres. There's an open issue to fix it: flake: TestReinitializeAgent internal#642

This PR will:

unblock Remove the in-memory database #15109
alleviate flake: ImDisk on Windows CLI internal#647

I tested caching by temporarily enabling cache upload on this PR: here's a run showing cache being used.

deansheather · 2025-05-20T01:45:18Z

.github/actions/setup-go-paths/action.yml

+      id: paths
+      uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
+      with:
+        script: |


I don't mind using the script here, but you can accomplish the same with bash by echoing to GITHUB_ENV and GITHUB_OUTPUT: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions

I chose to use the script to handle setting multi-line env variables. With bash, you have to be very careful with whitespace and indentation. It's error-prone.

deansheather · 2025-05-20T02:03:35Z

.github/workflows/ci.yaml


      - name: Checkout
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
        with:
          fetch-depth: 1

+      - name: Setup Go Paths
+        uses: ./.github/actions/setup-go-paths


Previously we were only doing this on Windows, should we do the same here?

This step is now used to standardize the Go cache location on disk so the "download cache step" knows where to put it. Without it, it's somewhat complicated to figure out the paths so it works cross-OS and if Go isn't installed.

deansheather · 2025-05-20T02:07:57Z

.github/workflows/ci.yaml

-    runs-on: ${{ matrix.os == 'ubuntu-latest' && github.repository_owner == 'coder' && 'depot-ubuntu-22.04-8' || matrix.os }}
+    # make sure to adjust NUM_PARALLEL_PACKAGES and NUM_PARALLEL_TESTS below
+    # when changing runner sizes
+    runs-on: ${{ matrix.os == 'ubuntu-latest' && github.repository_owner == 'coder' && 'depot-ubuntu-22.04-8' || matrix.os && matrix.os == 'macos-latest' && github.repository_owner == 'coder' && 'depot-macos-latest' || matrix.os == 'windows-2022' && github.repository_owner == 'coder' && 'depot-windows-2022-16' || matrix.os }}


This line is an abomination lmfao

.github/workflows/ci.yaml

deansheather · 2025-05-20T02:14:32Z

enterprise/coderd/workspaceagents_test.go

@@ -83,6 +84,10 @@ func TestBlockNonBrowser(t *testing.T) {
 func TestReinitializeAgent(t *testing.T) {
 	t.Parallel()

+	if runtime.GOOS == "windows" {
+		t.Skip("this test doesn't pass on Windows with Postgres")


Please open an internal ticket

the skip has already been merged separately: #17968, so I'm removing it from this PR

deansheather · 2025-05-20T02:18:37Z

.github/workflows/ci.yaml

+          # terraform gets installed in a random directory, so we need to normalize
+          # the path to the terraform binary or a bunch of cached tests will be
+          # invalidated. See scripts/normalize_path.sh for more details.
+          normalize_path_with_symlinks "$RUNNER_TEMP/sym" "$(dirname $(which terraform))"


I feel like it would be better if we changed the test to use system terraform if it's a suitable version, then we can just install it before starting the test.

Do you mean we should use the terraform download function that's built into Coder?

github-actions bot assigned hugodutka May 15, 2025

hugodutka force-pushed the hugodutka/test-go-pg-windows-macos-cache branch 23 times, most recently from c3f6634 to 291a7df Compare May 19, 2025 11:54

hugodutka changed the title ~~chore: run test-go-pg on macOS and Windows with test cache~~ chore: run test-go-pg on macOS and Windows in regular CI May 19, 2025

hugodutka force-pushed the hugodutka/test-go-pg-windows-macos-cache branch 5 times, most recently from 9c27aa8 to e321870 Compare May 19, 2025 14:07

hugodutka force-pushed the hugodutka/test-go-pg-windows-macos-cache branch 5 times, most recently from 3bcc4f4 to c8e68db Compare May 19, 2025 17:23

hugodutka marked this pull request as ready for review May 19, 2025 17:44

hugodutka requested a review from deansheather May 19, 2025 17:44

deansheather reviewed May 20, 2025

View reviewed changes

hugodutka force-pushed the hugodutka/test-go-pg-windows-macos-cache branch from c8e68db to 64646ed Compare May 22, 2025 10:13

hugodutka added 6 commits May 22, 2025 10:13

enable test-go-pg on macos and windows

fddaf7f

enable go test cache in test-go-pg

0edcb08

remove setup-imdisk

010d50d

remove nightly-gauntlet

6e7cc86

remove test-cli

09539ee

fix gen cancellation

2929c79

hugodutka force-pushed the hugodutka/test-go-pg-windows-macos-cache branch from 64646ed to 2929c79 Compare May 22, 2025 10:13

deansheather approved these changes May 22, 2025

View reviewed changes

hugodutka merged commit a0e229a into main May 22, 2025
35 of 37 checks passed

hugodutka deleted the hugodutka/test-go-pg-windows-macos-cache branch May 22, 2025 13:53

github-actions bot locked and limited conversation to collaborators May 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: run test-go-pg on macOS and Windows in regular CI #17853

chore: run test-go-pg on macOS and Windows in regular CI #17853

Uh oh!

hugodutka commented May 15, 2025 •

edited

Loading

Uh oh!

deansheather May 20, 2025

Uh oh!

hugodutka May 22, 2025

Uh oh!

deansheather May 20, 2025

Uh oh!

hugodutka May 22, 2025

Uh oh!

deansheather May 20, 2025

Uh oh!

Uh oh!

deansheather May 20, 2025

Uh oh!

hugodutka May 22, 2025 •

edited

Loading

Uh oh!

deansheather May 20, 2025

Uh oh!

hugodutka May 22, 2025

Uh oh!

Uh oh!

Uh oh!

chore: run test-go-pg on macOS and Windows in regular CI #17853

chore: run test-go-pg on macOS and Windows in regular CI #17853

Uh oh!

Conversation

hugodutka commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hugodutka May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hugodutka commented May 15, 2025 •

edited

Loading

hugodutka May 22, 2025 •

edited

Loading