-
Notifications
You must be signed in to change notification settings - Fork 891
feat: implement agent process management #9461
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
- An opt-in feature has been added to the agent to allow deprioritizing non coder-related processes for both CPU and memory. Non coder processes have their niceness set to 10 and their oom_score_adj set to 100
Why is it opt-in? Seems like a healthy default. Also, could remove a configuration knob. There's also implicit configuration in that the system can enable / disable priority management via the capabilities it gives the agent. |
Also: aiming to have a review for this by tomorrow afternoon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm confused about why we're processing all processes vs. just the agent PID itself? In theory niceness and oom_score is supposed to be relative—processes that want higher priority should be able to get it without knowledge of every other process.
The current approach whereby we're iterating all processes has scalability issues too. For example, if the agent runs in a large process namespace where it only controls a small number of processes, it would generate immense log spam.
agent/agent.go
Outdated
name := filepath.Base(proc.Name()) | ||
// If the process is prioritized we should adjust | ||
// it's oom_score_adj and avoid lowering its niceness. | ||
if slices.Contains(prioritizedProcs, name) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We want to specifically prioritize the agent and not other coder processes right? If I'm reading this code correctly it would treat coder server
and coder stat
the same as the agent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good catch, I don't see that as being a big deal but we can be more discriminate about which processes we want to prioritize by also parsing command arguments. WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not just check if its the current process?
This implementation was adopted from v1 since it is re-implementing functionality requested from a v1 customer. I imagine it was implemented this way because decreasing the niceness score of a process requires
I figured introducing this sort of functionality is safer to do opt-in, especially because interested parties only need to add a single env var to their template to enable it. If there's any unforeseen issues with the feature we avoid a serious regression by enabling it by default for every customer as opposed to the few who are specifically requesting this feature. Eventually I'd like to promote it to enabled by default once we're confident there aren't any surprises. |
SGTM |
agent/agent.go
Outdated
name := filepath.Base(proc.Name()) | ||
// If the process is prioritized we should adjust | ||
// it's oom_score_adj and avoid lowering its niceness. | ||
if slices.Contains(prioritizedProcs, name) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not just check if its the current process?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, but Ammar's comment about only prioritizing based on PID should be added before merge
In the future we may want to be able to prioritize other processes such as |
deprioritizing non coder-related processes for CPU by setting their
niceness level to 10.
CODER_PROC_MEMNICE_ENABLE
to a non-empty value.fixes #8517