Problem:

On Gateway 2022.2 RC, connecting to Coder over SSH hangs on detecting IDEs. Looking through the logs

idea.log

it appears that Gateway tests for the presence of an executable utility to find installed IDEs. This utility doesn't exist, but Gateway somehow concludes that is does. It then tries to execute this utility and fails.

Talking with JetBrains, their hypothesis is that our SSH server is not setting command return codes correctly when a PTY is allocated. Notes from Slack below:

Any chance something is overriding test shell builtin or unix utility on the host? Or some obscure SSH server setting results in all process exit codes being returned as 0?

Nikolay Kuznetsov
5 hours ago
Speaking of, is the SSH server some custom thing that is not OpenSSH?

Spike Curtis
45 minutes ago
It's our custom agent running https://github.com/gliderlabs/ssh

gliderlabs/ssh
Easy SSH servers in Golang
Website
https://godoc.org/github.com/gliderlabs/ssh
Stars
2628
Added by GitHub

Spike Curtis
38 minutes ago
test seems to be doing the right thing when I execute it through the regular ssh CLI:
spike@Codebook coder % ssh coder.dogfood test -f /home/coder/.cache/JetBrains/RemoteDev/remote-dev-worker/remote-dev-worker_8be48ed1f7d9d149a663842f0df2735b7433265d1654774596e54ce70af850f2
spike@Codebook coder % echo $?
1
spike@Codebook coder % ssh coder.dogfood test -f /home/coder/.cache/JetBrains/RemoteDev/remote-dev-worker/remote-dev-worker_7d32e4d2185f31ff477d51c73f533f4a312dfbd63132e75dc59ca2914b3965c4
spike@Codebook coder % echo $?
0

Nikolay Kuznetsov
18 minutes ago
Not sure if the repo you linked includes a default handler for "generic SSH server" use case. What if you specify -t or -T flags for ssh? Does that affect return value?

Spike Curtis
13 minutes ago
spike@Codebook coder % ssh -T coder.dogfood test -f /home/coder/.cache/JetBrains/RemoteDev/remote-dev-worker/remote-dev-worker_8be48ed1f7d9d149a663842f0df2735b7433265d1654774596e54ce70af850f2
spike@Codebook coder % echo $?
1
spike@Codebook coder % ssh -t coder.dogfood test -f /home/coder/.cache/JetBrains/RemoteDev/remote-dev-worker/remote-dev-worker_8be48ed1f7d9d149a663842f0df2735b7433265d1654774596e54ce70af850f2
Connection to coder.dogfood closed.
spike@Codebook coder % echo $?
0

Nikolay Kuznetsov
12 minutes ago
Found it I guess. By default Gateway requests a PTY for most commands to allow for SIGHUP-based process termination. Looks like your custom agent fails to properly handle exit codes in such cases.

Spike Curtis
< 1 minute ago
thanks for your help! I'll dig in some more on our side and see if I can sort this out