feat: modify coordinators to send errors and peers to log them #17467
Conversation
b7d0071 to
9cc149f
Compare
| // ReadyForHandshake error can occur during race conditions, where we send a ReadyForHandshake message, | ||
| // but the source has already disconnected from the tunnel by the time we do. So, just log at warning. | ||
| if strings.HasPrefix(resp.Error, ReadyForHandshakeError) { | ||
| c.logger.Warn(context.Background(), "coordination warning", slog.F("msg", resp.Error)) |
There was a problem hiding this comment.
This is a bit fragile in that it's special handling code for a specific error. But, I'm not sure we want to add more structure to the coordination protocol error messages for just one odd-duck error message.
Without this, many tests will flake due to dropping a benign error message.
@ethanndickson what do you think?
There was a problem hiding this comment.
Ah, I was indeed missing those log detected at level ERRORs...
All those test failures are in CLI integration tests. We already ignore error logs on coderdtest for this same reason, is it feasible to ignore agent error logs too (and only in integration tests)?
If not, this sounds fine to me as a warning.
There was a problem hiding this comment.
It's feasible, but I worry it's overbroad.
The ready for handshake "errors" aren't "errors" by the criteria we usually use for logs, which is that we definitely know something bad happened, rather than "something bad might have happened, or it might be fine" which is where we usually put WARN logs.
So, I think this behavior is more correct, I'm just not wild about the implementation. But, also not wild about alternative implementations that make the error vs warning explicit in the protocol.
There was a problem hiding this comment.
I think it's fine to write as a one-off. If we ever need to communicate more than one type of warning using the protocol, then maybe we reconsider.
There was a problem hiding this comment.
Yeah, that's what I was thinking. If we have to start special casing more than one, it will be worth while to make it explicit in the protocol.
Adds support to our coordinator implementations to send Error updates before disconnecting clients.
I was recently debugging a connection issue where the client was getting repeatedly disconnected from the Coordinator, but since we never send any error information it was really hard without server logs.
This PR aims to correct that, by sending a CoordinateResponse with
Errorset in cases where we disconnect a client without them asking us to.It also logs the error whenever we get one in the client controller.