Skip to content

chore: shutdown provisioner should stop waiting on client #13118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 3, 2024
Merged

chore: shutdown provisioner should stop waiting on client #13118

merged 3 commits into from
May 3, 2024

Conversation

Emyrk
Copy link
Member

@Emyrk Emyrk commented May 1, 2024 •
edited
Loading

provisionerd does not respond to ctrl+c if no coderd exits.

Run go run main.go provisionerd start without a coderd. After a second or two, hit ctrl +c. The process will not terminate. The issue is we call Shutdown for a graceful closure.

Currently:

  • connect() does not terminate on shutdown. Only on Close
  • Shutdown does not call close. The caller needs to call Shutdown and then Close. Updated the cli to do this, although it is not intuitive imo.
  • acquireLoop blocks on a client, but if no client is ever established, the shutdown never gets checked, and this blocks forever.

This PR calls Close after Shutdown and makes p.client() return a nil client during shutdown.

I'd like to see provisionerd's Shutdown call Close.

@Emyrk Emyrk requested a review from spikecurtis May 1, 2024 19:39
spikecurtis
spikecurtis requested changes May 2, 2024
View reviewed changes
Copy link
Contributor

@spikecurtis spikecurtis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a unit test?

@Emyrk
Copy link
Member Author

Emyrk commented May 2, 2024

@spikecurtis test added:

coder/provisionerd/provisionerd_test.go

Lines 602 to 630 in 418c4dd

t.Run("ShutdownNoCoderd", func(t *testing.T) {
t.Parallel()
done := make(chan struct{})
t.Cleanup(func() {
close(done)
})
connectAttemptedClose := sync.Once{}
connectAttempted := make(chan struct{})
server := createProvisionerd(t, func(ctx context.Context) (proto.DRPCProvisionerDaemonClient, error) {
// This is the dial out to Coderd, which in this unit test will always fail.
connectAttemptedClose.Do(func() { close(connectAttempted) })
return nil, fmt.Errorf("client connection always fails")
}, provisionerd.LocalProvisioners{
"someprovisioner": createProvisionerClient(t, done, provisionerTestServer{}),
})
// Wait for at least 1 attempt to connect to ensure the connect go routine
// is running.
require.Condition(t, closedWithin(connectAttempted, testutil.WaitShort))
// The test is ensuring this Shutdown call does not block indefinitely.
// If it does, the context will return with an error, and the test will
// fail.
shutdownCtx := testutil.Context(t, testutil.WaitShort)
err := server.Shutdown(shutdownCtx, true)
require.NoError(t, err, "shutdown did not unblock. Failed to close the server gracefully.")
require.NoError(t, server.Close())
})

@Emyrk Emyrk requested a review from spikecurtis May 2, 2024 16:30
spikecurtis
spikecurtis approved these changes May 3, 2024
View reviewed changes
Copy link
Contributor

@spikecurtis spikecurtis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

provisionerd/provisionerd_test.go

// Wait for at least 1 attempt to connect to ensure the connect go routine
// is running.
require.Condition(t, closedWithin(connectAttempted, testutil.WaitShort))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just FYI, we have testutil.RequireRecvCtx() which is what I usually use for this sort of thing.

@Emyrk Emyrk merged commit 09f00c0 into main May 3, 2024
25 checks passed
@Emyrk Emyrk deleted the stevenmasley/provisioner_close branch May 3, 2024 15:15
@github-actions github-actions bot locked and limited conversation to collaborators May 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@spikecurtis spikecurtis spikecurtis approved these changes

Assignees

@Emyrk Emyrk

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Emyrk @spikecurtis