Skip to content

Update to v2.8.0 breaks connection to coder #12057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
MrPeacockNLB opened this issue Feb 7, 2024 · 11 comments · Fixed by #12060
Closed

Update to v2.8.0 breaks connection to coder #12057

MrPeacockNLB opened this issue Feb 7, 2024 · 11 comments · Fixed by #12060
Labels
s0 Major regression, all-hands-on-deck to fix

Comments

@MrPeacockNLB
Copy link
Contributor

Description

After update to the latest release v2.8.0 suddenly the communication between workspace and coder instance is not possible.

image

There were a ton of yamux errors in workspace logs.

What was changed

We are using coder with external provisionerd in an Kubernetes setup. The only change was to raise both versions from v2.7.2 to v2.8.0

What is expected

A update without a breaking change should work
@cdr-bot cdr-bot bot added the bug label Feb 7, 2024
@kylecarbs
Copy link
Member

Woah, this shouldn't be happening.

If you rebuild the workspace does it work?

@bpmct bpmct added the s0 Major regression, all-hands-on-deck to fix label Feb 7, 2024
@MrPeacockNLB
Copy link
Contributor Author

I just restarted 2 workspaces. Both the same error. Let me try to create a new one from scratch to see if it work

@kylecarbs
Copy link
Member

Is there any chance something with your infrastructure changed? Or are you confident it's on the Coder side?

@MrPeacockNLB
Copy link
Contributor Author

nothing has changed. Just updated helm charts to use newest tag

@kylecarbs
Copy link
Member

Got it. We're working on this as a P1... will keep you posted.

@bpmct
Copy link
Member

bpmct commented Feb 7, 2024 •
edited
Loading

@MrPeacockNLB We just sent you an email if you want to set up a call to troubleshoot live

@spikecurtis
Copy link
Contributor

That error comes from the websocket layer --- the limit seems to be fairly arbitrary, but set to that by default. I wonder if they have an unusually large startup script or something.

@MrPeacockNLB
Copy link
Contributor Author

yes, our startup script is huge as it takes all the logic.

@MrPeacockNLB
Copy link
Contributor Author

sadly I am unable to verify if a workspace with a small startup_script fixes the problem. This is a chicken egg problem as we were using a coder workspace to publish a new template version.

@spikecurtis spikecurtis mentioned this issue Feb 7, 2024
Merged
spikecurtis added a commit that referenced this issue Feb 7, 2024
@spikecurtis
fix: allow startup scripts larger than 32k (#12060)
151aaad 
Fixes #12057 and adds a regression test.
bpmct pushed a commit that referenced this issue Feb 7, 2024
@spikecurtis @bpmct
fix: allow startup scripts larger than 32k (#12060)
f519db8 
Fixes #12057 and adds a regression test.
@bpmct
Copy link
Member

bpmct commented Feb 7, 2024

We just released v2.8.1 with this fix!

@MrPeacockNLB
Copy link
Contributor Author

It is working again. Tested this just now.

Thank you all Coder Team! This was a great support. Never seen this before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
s0 Major regression, all-hands-on-deck to fix
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: allow startup scripts larger than 32k coder/coder
4 participants
@spikecurtis @kylecarbs @bpmct @MrPeacockNLB