fix: close SSH sessions bottom-up if top-down fails #14678
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @spikecurtis and the rest of your teammates on |
mafredri
reviewed
Sep 16, 2024
d9dc47a to
21ee4de
Compare
mafredri
approved these changes
Sep 17, 2024
21ee4de to
a3d0c5b
Compare
a3d0c5b to
7dd56e0
Compare
Merge activity
|
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes https://github.com/coder/customers/issues/669
In
coder sshwe normally attempt to tear stuff down top to bottom, so that SSH stuff like remote-forwards are cleaned up nicely, tailnet Coordination gets a disconnect, etc.But, TCP timeouts can be very long (72 hours currently for SSH), and so if network connectivity down, we can effectively deadlock trying to tear down the remote-forward state, which involves sending an SSH command message and doesn't time out independently of the underlying TCP connection.
This PR introduces a "graceful shutdown" timeout for the upper layers of our SSH stuff to finish closing. If they haven't closed in 5 seconds, we shut down the agent connection, which cascades bottom-up.