fix: fix goroutine leak in log streaming over websocket #15709
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
e379c84 to
2ae0a74
Compare
mafredri
approved these changes
Dec 2, 2024
coderd/workspaceagents.go
Outdated
| ctx, wsNetConn := codersdk.WebsocketNetConn(ctx, conn, websocket.MessageText) | ||
| defer wsNetConn.Close() // Also closes conn. | ||
| encoder := wsjson.NewEncoder[[]codersdk.WorkspaceAgentLog](conn, websocket.MessageText) | ||
| defer encoder.Close(websocket.StatusGoingAway) |
There was a problem hiding this comment.
Why do we use going away instead of normal closure? It's kind of an error state and a divergence from before (i.e. status from calling wsNetConn.Close()).
There was a problem hiding this comment.
I chose GoingAway because one likely reason for the server closing the connection is that it's shutting down. We could also be closing because there is a new build and this agent is no longer current.
If we actually used status codes for anything we'd want to send different codes in these cases, but we don't. I can change it to match the old wsNetConn for consistency.
| @@ -767,7 +758,7 @@ func (api *API) derpMapUpdates(rw http.ResponseWriter, r *http.Request) { | |||
| err := ws.Ping(ctx) | |||
| cancel() | |||
| if err != nil { | |||
| _ = nconn.Close() | |||
| _ = ws.Close(websocket.StatusGoingAway, "ping failed") | |||
codersdk/wsjson/decoder.go
Outdated
|
|
||
| // nolint: revive // complains that Encoder has the same function name | ||
| func (d *Decoder[T]) Close() error { | ||
| err := d.conn.Close(websocket.StatusGoingAway, "") |
There was a problem hiding this comment.
Same as previously, why not use normal closure? Also, why not take status like encoder for consistency?
There was a problem hiding this comment.
I've switched to StatusNormalClosure.
Here I don't want to take the websocket status like Encoder because it's useful for the Decoder to implement io.Closer to more easily fit into existing code.
2ae0a74 to
76911bb
Compare
Merge activity
|
stirby
pushed a commit
that referenced
this pull request
Dec 3, 2024
fixes #14881 Our handlers for streaming logs don't read from the websocket. We don't allow the client to send us any data, but the websocket library we use requires reading from the websocket to properly handle pings and closing. Not doing so can [can cause the websocket to hang on write](coder/websocket#405), leaking go routines which were noticed in #14881. This fixes the issue, and in process refactors our log streaming to a encoder/decoder package which provides generic types for sending JSON over websocket. I'd also like for us to upgrade to the latest https://github.com/coder/websocket but we should also upgrade our tailscale fork before doing so to avoid including two copies of the websocket library. (cherry picked from commit 148a5a3)
stirby
pushed a commit
that referenced
this pull request
Dec 11, 2024
fixes #14881 Our handlers for streaming logs don't read from the websocket. We don't allow the client to send us any data, but the websocket library we use requires reading from the websocket to properly handle pings and closing. Not doing so can [can cause the websocket to hang on write](coder/websocket#405), leaking go routines which were noticed in #14881. This fixes the issue, and in process refactors our log streaming to a encoder/decoder package which provides generic types for sending JSON over websocket. I'd also like for us to upgrade to the latest https://github.com/coder/websocket but we should also upgrade our tailscale fork before doing so to avoid including two copies of the websocket library. (cherry picked from commit 148a5a3)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #14881
Our handlers for streaming logs don't read from the websocket. We don't allow the client to send us any data, but the websocket library we use requires reading from the websocket to properly handle pings and closing. Not doing so can can cause the websocket to hang on write, leaking go routines which were noticed in #14881.
This fixes the issue, and in process refactors our log streaming to a encoder/decoder package which provides generic types for sending JSON over websocket.
I'd also like for us to upgrade to the latest https://github.com/coder/websocket but we should also upgrade our tailscale fork before doing so to avoid including two copies of the websocket library.