Skip to content

tailnet coordinator periodically disconnects clients & agents #7428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
spikecurtis opened this issue May 5, 2023 · 2 comments · Fixed by #7430
Closed

tailnet coordinator periodically disconnects clients & agents #7428

spikecurtis opened this issue May 5, 2023 · 2 comments · Fixed by #7430
Assignees
@spikecurtis
Labels
s1 Bugs that break core workflows. Only humans may set this.

Comments

@spikecurtis
Copy link
Contributor

Agent disconnects started increasing substantially with #7345

coordinator-agent

Logs: https://app.datadoghq.com/logs?query=%40caller%3A%2Fhome%2Frunner%2Fwork%2Fcoder%2Fcoder%2Ftailnet%2Fcoordinator.go%5C%3A%2A%20%40msg%3A%28%22unable%20to%20read%20agent%20update%3B%20closed%20conn%3F%22%20OR%20%22unable%20to%20read%20client%20update%3B%20closed%20conn%3F%22%20OR%20%22could%20not%20write%20nodes%20to%20connection%22%29&agg_q=%40fields.error%2C%40fields.error&analyticsOptions=%5B%22bars%22%2C%22dog_classic%22%2Cnull%2Cnull%5D&cols=host%2Cservice&index=%2A&messageDisplay=inline&sort_m=%2C&sort_t=%2C&stream_sort=desc&top_n=10%2C10&top_o=top%2Ctop&viz=toplist&x_missing=true%2Ctrue&from_ts=1682961720000&to_ts=1683220920000&live=false
@spikecurtis spikecurtis added the s1 Bugs that break core workflows. Only humans may set this. label May 5, 2023
@spikecurtis spikecurtis self-assigned this May 5, 2023
@spikecurtis
Copy link
Contributor Author

spikecurtis commented May 5, 2023 •
edited
Loading

So, it turns out https://github.com/nhooyr/websocket is bugged when we use SetWriteDeadline --- the next write will often fail. Prior to #7345 we only called this on client connections, but now we use it in both directions, so we're seeing agents get disconnected now. We probably didn't notice the client disconnections because a certain amount of client disconnects are of course expected.

Fix incoming.

@spikecurtis spikecurtis mentioned this issue May 5, 2023
Merged
@spikecurtis
Copy link
Contributor Author

I have to wonder whether the client disconnections that were present even before #7345 are the underlying cause for some of the flakiness we have seen in SSH, JetBrains Gateway, etc.

@matifali matifali added the bug label May 5, 2023
@matifali matifali added the bug label Jun 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spikecurtis spikecurtis

Labels
s1 Bugs that break core workflows. Only humans may set this.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: agent disconnects from coordinator coder/coder
2 participants
@spikecurtis @matifali