Skip to content

fix: wait for provisioner daemon jobs to finish after sigterm #14466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

ethanndickson
Copy link
Member

@ethanndickson ethanndickson commented Aug 28, 2024

Fixes #14433.

The function for shutting down a provisioner daemon has the signature:

func (p *Server) Shutdown(ctx context.Context, cancelActiveJob bool) error

The coder server code for handling graceful provisioner shutdowns was correct:

err := shutdownWithTimeout(func(ctx context.Context) error {
// We only want to cancel active jobs if we aren't exiting gracefully.
return provisionerDaemon.Shutdown(ctx, !waitForProvisionerJobs)
}, timeout)

Whilst the matching code for graceful shutdowns of independent provisioner daemons (coder provisionerd start) was not:

err = srv.Shutdown(ctx, waitForProvisionerJobs)

This wasn't caught by tests as I don't believe we have any clitests for provisionerd. The existing tests call the shutdown function directly.

Copy link
Member Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @ethanndickson and the rest of your teammates on Graphite Graphite

@ethanndickson ethanndickson marked this pull request as ready for review August 28, 2024 10:13
Copy link

@cdr-bot cdr-bot bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is a hotfix and has been automatically approved.

  • ✅ Base is main or release branch
  • ✅ Has hotfix label
  • ✅ Head is from coder/coder
  • ✅ Less than 100 lines

@ethanndickson ethanndickson merged commit c597c92 into main Aug 28, 2024
41 checks passed
@ethanndickson ethanndickson deleted the 08-28-fix_wait_for_provisioner_daemon_jobs_to_finish_after_sigterm branch August 28, 2024 10:16
@github-actions github-actions bot locked and limited conversation to collaborators Aug 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"Graceful" shutdown with SIGTERM appears to interrupt Teraform provider
1 participant