This is really odd. I'll test this out!

@kylecarbs did you have a chance to look at this? No worries if not, I was planning on taking a look at this tomorrow, so just checking in.

Nah, I wasn't able to look at it. Feel free to poke around!

@mtm20176 does this still happen for you? It seems to work for me on 2022.1.3.

@mtm20176 @kylecarbs I'm still reproducing this with 2022.1.3
2022-05-12 01:00:40,749 [ 135] INFO - #c.i.i.StartupUtil - IDE: JetBrains Gateway (build #GW-221.5080.246, 20 Apr 2022 18:54) 2022-05-12 01:00:40,749 [ 135] INFO - #c.i.i.StartupUtil - OS: Windows 10 (10.0, amd64) 2022-05-12 01:00:40,752 [ 138] INFO - #c.i.i.StartupUtil - JRE: 11.0.14.1+1-b2043.25 (JetBrains s.r.o.) 2022-05-12 01:00:40,753 [ 139] INFO - #c.i.i.StartupUtil - JVM: 11.0.14.1+1-b2043.25 (OpenJDK 64-Bit Server VM) 2022-06-06 22:21:42,377 [ 31479] SEVERE - #c.j.g.w.MultistagePanelContainer - File C:\Users\ifaur\AppData\Local\JetBrains\Toolbox\apps\JetBrainsGateway\ch-0\221.5787.16\lib\remote-dev-workers\remote-dev-worker-linux-amd64: chown /home/coder/.cache/JetBrains/RemoteDev/remote-dev-worker/remote-dev-worker_7d32e4d2185f31ff477d51c73f533f4a312dfbd63132e75dc59ca2914b3965c4.39064324895700.tmp: operation not permitted com.intellij.ssh.SftpChannelPermissionDeniedException: File C:\Users\ifaur\AppData\Local\JetBrains\Toolbox\apps\JetBrainsGateway\ch-0\221.5787.16\lib\remote-dev-workers\remote-dev-worker-linux-amd64: chown /home/coder/.cache/JetBrains/RemoteDev/remote-dev-worker/remote-dev-worker_7d32e4d2185f31ff477d51c73f533f4a312dfbd63132e75dc59ca2914b3965c4.39064324895700.tmp: operation not permitted at com.intellij.ssh.impl.sshj.channels.SshjSftpChannel.uploadFileOrDir(SshjSftpChannel.kt:175) at com.jetbrains.gateway.ssh.GoSshCommandExecutorDeployer.deploy(GoSshCommandExecutorDeployer.kt:61) at com.jetbrains.gateway.ssh.SshCommandsExecutor$Companion.create(SshCommandsExecutor.kt:56) at com.jetbrains.gateway.ssh.panels.ChooseHostPanel$onGoingToLeave$executor$1.invokeSuspend(ChooseHostPanel.kt:279) at com.jetbrains.gateway.ssh.panels.ChooseHostPanel$onGoingToLeave$executor$1.invoke(ChooseHostPanel.kt) at com.jetbrains.rd.framework.util.LifetimeCoroutineUtilKt$withContext$2.invokeSuspend(LifetimeCoroutineUtil.kt:95) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: net.schmizz.sshj.sftp.SFTPException: chown /home/coder/.cache/JetBrains/RemoteDev/remote-dev-worker/remote-dev-worker_7d32e4d2185f31ff477d51c73f533f4a312dfbd63132e75dc59ca2914b3965c4.39064324895700.tmp: operation not permitted at net.schmizz.sshj.sftp.Response.error(Response.java:140) at net.schmizz.sshj.sftp.Response.ensureStatusIs(Response.java:133) at net.schmizz.sshj.sftp.Response.ensureStatusPacketIsOK(Response.java:125) at net.schmizz.sshj.sftp.SFTPEngine.setAttributes(SFTPEngine.java:177) at net.schmizz.sshj.sftp.SFTPClient.setattr(SFTPClient.java:148) at com.intellij.ssh.impl.sshj.channels.SshjSftpChannel.persistExecutable(SshjSftpChannel.kt:76) at com.intellij.ssh.impl.sshj.channels.SshjSftpChannel.uploadFileOrDir(SshjSftpChannel.kt:56) ... 14 more

I'll have to try this from Windows, that's where it appears to happen!

This might be a dupe of #1386 --- I'm reproducing this problem on macOS, and seeing complaints about no $HOME in the logs.

2022-06-06 15:20:00,101 [ 243317]   WARN - #c.j.g.s.GoSshCommandsExecutor - An error occurred while execution command: 'installed-ides'
Exit code: 1
Stderr: 'mesg: ttyname failed: Inappropriate ioctl for device
panic: $HOME is not defined

goroutine 1 [running]:
jetbrains.com/remote-dev-worker/utils.RunCommandFromArgs({0xc00009e010, 0x40bbc7, 0x0}, {0xc000088e30, 0x14, 0x14})
	/src/utils/commands.go:30 +0x73
main.main()
	/src/main.go:17 +0x254
'
2022-06-06 15:20:00,104 [ 243320] SEVERE - #c.j.g.s.p.LocateRemoteProjectPanel - $HOME is not defined

Details:
An error occurred while execution command: 'installed-ides'
Exit code: 1
java.lang.IllegalStateException: $HOME is not defined

Details:
An error occurred while execution command: 'installed-ides'
Exit code: 1
	at com.jetbrains.gateway.ssh.GoSshCommandsExecutor.anErrorOccurred(GoSshCommandsExecutor.kt:22)
	at com.jetbrains.gateway.ssh.GoSshCommandsExecutor.getInstalledIDEs(GoSshCommandsExecutor.kt:197)
	at com.jetbrains.gateway.ssh.panels.LocateRemoteProjectPanel$getComponent$1$3$2$1.run(LocateRemoteProjectPanel.kt:189)
	at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:295)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
	at java.base/java.lang.Thread.run(Thread.java:829)
2022-06-06 15:20:00,106 [ 243322] SEVERE - #c.j.g.s.p.LocateRemoteProjectPanel - JetBrains Gateway 2022.1.1  Build #GW-221.5080.246
2022-06-06 15:20:00,107 [ 243323] SEVERE - #c.j.g.s.p.LocateRemoteProjectPanel - JDK: 11.0.14.1; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o.
2022-06-06 15:20:00,107 [ 243323] SEVERE - #c.j.g.s.p.LocateRemoteProjectPanel - OS: Mac OS X

Did some debugging on the client side and these are the attributes that Gateway tries to set:

path = "/home/coder/.cache/JetBrains/RemoteDev/remote-dev-worker/remote-dev-worker_7d32e4d2185f31ff477d51c73f533f4a312dfbd63132e75dc59ca2914b3965c4.4397788083400.tmp"
attrs = {FileAttributes@15927} "[size=1347584;uid=1000,gid=1000;mode=[mask=100744];atime=1654545923,mtime=1654545923;]"
 mode = {FileMode@18435} "[mask=100744]"
  mask = 33252
  type = {FileMode$Type@18438} "REGULAR"
  perms = {HashSet@18439}  size = 6
   0 = {FilePermission@18442} "USR_W"
   1 = {FilePermission@18443} "OTH_R"
   2 = {FilePermission@18444} "GRP_R"
   3 = {FilePermission@18445} "USR_RWX"
   4 = {FilePermission@18446} "USR_R"
   5 = {FilePermission@18447} "USR_X"
 mask = 15
 size = 1347584
 uid = 1000
 gid = 1000
 atime = 1654545923
 mtime = 1654545923
 ext = {HashMap@18436}  size = 0

The uid and gid are correct:

coder@rocky:~$ id coder
uid=1000(coder) gid=1000(coder) groups=1000(coder),999(docker)`

I was able to connect just fine from MacOS, Gateway 2022.1.1, and the do-linux template, as well as gcp-linux template after #2150.

@mtm20176 @fioan89 can you try again with the latest gcp-linux template, and if it's not working include your OS so that we can narrow this down? I fixed the $HOME issue on that template and am wondering if that's the root cause...

@spikecurtis Indeed I was able to connect without issues, I could do it since yesterday. Thx.

FTW! Can we close this or is @mtm20176 still having trouble?

From @fioan89 's attribute trace, I'm trying to understand what permissions it is trying to set:

It says mask=100744 and mask=33252 which are consistent if the first is interpreted as octal. A permission like 744 is pretty standard on Linux: -rwxr--r--, but what's going on with the leading 100? I think Linux uses 4 octal digits for permission since there are some extra bits above the user/group/all stuff, but not an extra 12 bits. Could this be the root issue?

Interesting. This https://www.mkssoftware.com/docs/man1/chmod.1.asp suggests that 100000 (octal) is the "temporary file" attribute, and as far as I can tell, it only exists on Windows.

This is really starting to smell like a cross-platform bug in JetBrains: the file they are messing with ends with .tmp, so they're pretty clearly creating a temporary file. Client code running on Windows decides it should set the temporary file attribute, but this attribute doesn't exist on the remote side of the connection (Linux).

I just tried and ran into issues.

Version:
JetBrains Gateway 2022.2 EAP
Build #GW-222.3048.1, built on June 13, 2022
Runtime version: 17.0.3+7-b469.12 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Windows 11 10.0
GC: G1 Young Generation, G1 Old Generation
Memory: 512M
Cores: 6

I installed 0.7.3 cli and authenticated, and ssh coder.jetbrains will work for me on a terminal. When I try to use Gateway the connection will hang for a while with the following configuration:

Then eventually will time out with:

Gateway logs: gateway-logs-20220622-14000317181305963056673646.zip

@mafredri suggested that a workaround could be to use the new SSH parser (maybe fixes on macos)

Closing this based on our discussion in GRooming

Where we ended up with this is that 2022.1.x doesn't work with a Windows client connecting to a Linux server due to a Gateway bug. JetBrains doesn't seem interested in doing a patch, and they've rewritten the part of the code that had this bug for 2022.2.

So, we're just going to have to live with this and focus on getting 2022.2 working with Coder.