Jetbrains plugin clients are reporting both 0 and 1 for workspace session counts with workspace-usage experiment enabled #15176
Assignees
Labels
bug risk
Prone to bugs
observability
Issues related to observability (metrics, dashboards, alerts, opentelemetry)
Comments
|
When connecting with the latest gateway version and coder plugin I'm seeing the following behavior:
So there's 2 problems to solve:
|
|
@code-asher Would you be able to help me track down this extra "IDE connection" being made in the jetbrains plugin? Basically there's a few places in the plugin we make SSH connections where we pass Here's the gateway logs shows the two IDE connections using the non-background hostname: One thing I can notice is that the two have different ports in the targetted hostname:
|
|
After some more investigation into the underlying ssh processes going on I've noticed the following: After opening IDE to workspace: Unfortunately these 4 connections do not get closed when I close the IDE window. As long as the Gateway landing window is still open it seems these SSH connections are held open indefinitely. So even if we correctly use the background hostname for 1 of the 2 connections, it seems like Jetbrains Gateway will keep the old IDE SSH connection around for a while (I'm testing to see just how long right now). |
This was referenced Oct 25, 2024
|
It's also possible the issue is on our end and we are failing to close the ssh process properly, maybe something here: https://github.com/coder/coder/blob/main/cli/ssh.go#L318 |
|
I think it might actually be the IDE keeping the connections open (perhaps intentionally)? Strangely enough, it doesn't do this for other SSH connections. I wonder if it's related to the |
|
Thanks for looking into that and helping with the search. I've also been building and rebuilding the plugin locally so I can do some AB testing and I've narrowed down both SSH connections with I'm not sure if we have control over the layer that's making the client and holding open these SSH connections. Something in the SSH config may be able to help us here. |
|
Here's a client log that is showing constant reconnections with the main hostname, not sure if this is exactly related but could be a sign that we possibly aren't closing a connection somewhere? https://gist.github.com/f0ssel/3f52cc8dd078a4079fc2b4aaf16d161d |
|
Closing in favor of coder/internal#179 now that we better understand the root cause. |
f0ssel
added a commit
that referenced
this issue
Oct 29, 2024
Part of #15176 I originally kept this the same because I wanted to be conservative about when we start dropping activity, but this is proving to be a problem when using `coder ssh` with `--usage-app=disabled`. Because the workspace agent still counts this as a connection (I think it still should so it's counted somewhere) but not as a SSH / IDE session. This leads to background ssh tasks that want to be untracked still continuing to bump activity when it shouldn't. This makes it so we have to have an explicit session to bump activity.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If you query the prometheus endpoint while connected to a workspace via the jetbrains plugin you can see the query sometimes return
0and sometimes return1. You would expect a solid1no matter when in the interval you query.I think this may be caused by 2 separate reporting loops and possibly a bug in the query where we are selecting the most recent instead of a sum of the most recent.
The text was updated successfully, but these errors were encountered: