I'm going with exposing agent metrics via Prometheus endpoint.

Status update:

We have agent connection/latencies/session stats exposed via Prometheus. The next step would be collecting and exposing details related to agent timeouts.

Plan for this week:

Implement a metrics collector in coderd to aggregate metrics from agents. Let's collect magicsock metrics first.

• coderd: adjust /report-stats API endpoint to receive metrics
• agent: send magicsock metrics
• coderd: save metrics to Prometheus registry

Hey @mafredri! Did you identify any code places where we can inject extra metrics/counters to debug agent timeouts easier? I'm wondering if there are any gaps that could be addressed here.

@mtojek not really. I still haven’t managed to isolate the problem and can’t really say what metric would help. Although, agent metrics is only one side of the coin, knowing what the client sees could help in such situations. Something I’m doing now is adding logging on the client side. But we probably shouldn’t be sending client metrics to the server, at least not normally.

There is a significant problem with logging on the agent side. Admins have to ask users to copy, or at least review their logs for issues. We definitely need something centralized, ideally a sink for logs, but I wouldn't mind some extra metrics, even vague ones like agent_vscode_network_timeout or agent_ssh_bad_peer_cerfiticate.

Battle plan:

• feat: Collect agent SSH metrics #7584 Add metrics to the SSH server wrapper, and possibly to https://github.com/coder/ssh.
  • distinguish: PTY, coder ssh, ssh coder (stdio)
  • add metrics to the ReconnectingPTY
• agent: Add support for keep-alive SSH messages #7581
• Enable back sshMaxTimeout once agent: Add support for keep-alive SSH messages #7581 is implemented.

Related #4680,

I'm going to detach #7581 from the plan and keep it as a separate issue, hence resolving it.