Skip to content

feat: add SIGQUIT/SIGTRAP handler for the CLI #5665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 11, 2023
Merged

Conversation

deansheather
Copy link
Member

@deansheather deansheather commented Jan 11, 2023

The default SIGQUIT handler in Golang dumps all goroutine stacktraces to stderr and then exits the program.

This changes the behavior so it dumps all goroutine stacktraces to stderr and a file (with the date) to $HOME, then exits. For the SIGTRAP handler, it does the same thing but doesn't exit.


// Write to a well-known file.
dir, err := os.UserHomeDir()
if err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why write to home? I'd rather just have these files in my temp dir.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On darwin the temp dir is a very long random path (not in /tmp) so it's difficult to find the file afterwards, especially if you can't read stderr to see the filename (which is the case a lot of the time).

It seems weird to have different behavior on different platforms, and this is a rare debug step so I think this is fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handler is now only applied to server and agent, and exits on SIGQUIT again

Copy link
Member

@mafredri mafredri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unclear to me why we want to do this and what problem we're addressing. Especially the part where we continue running after SIGQUIT, that seems like a feature we wouldn't want in the coder binary, perhaps only in server and agent.

At least as a user I would want the process to exit on SIGQUIT.

Have we evaluated other options, like setting export GOTRACEBACK=crash before running the command? This will both output to stderr and produce a core file.

One thing we lose by only outputting runtime.Stack is the register values that are usually produced when core is dumped. It seems this signal handling also interferes with GOTRACEBACK=crash as in it's no longer possible to get a proper core dump via SIGQUIT.

if err != nil {
dir = os.TempDir()
}
fpath := filepath.Join(dir, fmt.Sprintf("coder-agent-%s.dump", time.Now().Format("2006-01-02T15:04:05.000Z")))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these files are usually written to the current working directory, is there a reason we don't want to do that here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These files need to be in a well known location otherwise they're difficult to find when people are trying to send us debug logs. Since coder agent (which this is intended for) is a daemon, writing the working directory will mean the user will have to look in multiple places for the dump.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can understand the use-case for workspaces, I think we have more flexibility in dumping into home there. Should we at least move it to the agent cmd in that case?

The loss register values and supporting GOTRACEBACK is unfortunate, though. We could at least enable GOTRACEBACK by conditionally starting this routine only if it's unset.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disabled listening for SIGQUIT if GOTRACEBACK=crash, is that good?

@mafredri mafredri changed the title feat: add SIQGUIT handler for the CLI feat: add SIGQUIT handler for the CLI Jan 11, 2023
@deansheather
Copy link
Member Author

I will change the behavior to listen for both SIGQUIT and SIGTRAP, and to os.Exit(1) on SIGTRAP.

@mafredri
Copy link
Member

I will change the behavior to listen for both SIGQUIT and SIGTRAP, and to os.Exit(1) on SIGTRAP.

Could we make it the other way around? SIGTRAP seems like a better candidate for leaving the process running, as opposed to SIGQUIT which is supposed to quit the program.

@deansheather
Copy link
Member Author

err i meant SIGQUIT oops

@deansheather deansheather requested a review from mafredri January 11, 2023 13:57
@deansheather deansheather changed the title feat: add SIGQUIT handler for the CLI feat: add SIGQUIT/SIGTRAP handler for the CLI Jan 11, 2023
Copy link
Member

@mafredri mafredri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turned out great. 👍🏻

@deansheather deansheather enabled auto-merge (squash) January 11, 2023 16:18
@deansheather deansheather merged commit e72a2ad into main Jan 11, 2023
@deansheather deansheather deleted the dean/sigquit branch January 11, 2023 16:22
@github-actions github-actions bot locked and limited conversation to collaborators Jan 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants