Skip to content

Add prometheus metric for in flight requests #17212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
deansheather opened this issue Apr 2, 2025 · 3 comments · Fixed by #17362
Closed

Add prometheus metric for in flight requests #17212

deansheather opened this issue Apr 2, 2025 · 3 comments · Fixed by #17362
Assignees

Comments

@deansheather
Copy link
Member

deansheather commented Apr 2, 2025

Add a prometheus gauge metric that tracks the amount of in flight requests. Using a middleware, we would increment the gauge, and decrement it with a deferred function.

This metric will be useful to detect strange behavior (like websocket spam) or something, which will be super useful to alert on especially once we have #16904.

If it's easy, this should also label by route pattern e.g. route="/api/v2/workspaceagents/:id or something. Don't include IDs in the metrics to avoid ballooning labels.

@sreya
Copy link
Collaborator

sreya commented Apr 2, 2025

Consider adding a separate gauge for active websocket connections

@ibetitsmike
Copy link
Contributor

ibetitsmike commented Apr 2, 2025

AFAICT there's already code for doing both:

requestsConcurrent := factory.NewGauge(prometheus.GaugeOpts{

requestsConcurrent := factory.NewGauge(prometheus.GaugeOpts{
		Namespace: "coderd",
		Subsystem: "api",
		Name:      "concurrent_requests",
		Help:      "The number of concurrent API requests.",
	})
websocketsConcurrent := factory.NewGauge(prometheus.GaugeOpts{
		Namespace: "coderd",
		Subsystem: "api",
		Name:      "concurrent_websockets",
		Help:      "The total number of concurrent API websockets.",
	})

used in:

if httpapi.IsWebsocketUpgrade(r) {

// We want to count WebSockets separately.
if httpapi.IsWebsocketUpgrade(r) {
	websocketsConcurrent.Inc()
	defer websocketsConcurrent.Dec()

	dist = websocketsDist
} else {
	requestsConcurrent.Inc()
	defer requestsConcurrent.Dec()

	dist = requestsDist
	distOpts = []string{method}
}

@ibetitsmike
Copy link
Contributor

the labeling part seems to be missing, if we think the base is OK I can work on extending this with a path label

ibetitsmike added a commit that referenced this issue Apr 22, 2025
…ests (cherry-pick #17362) (#17493)

Cherry-picked feat: add path & method labels to prometheus metrics for
current requests (#17362)

Closes: #17212

Co-authored-by: Michael Suchacz <203725896+ibetitsmike@users.noreply.github.com>
ibetitsmike added a commit that referenced this issue Apr 22, 2025
…ests (cherry-pick #17362) (#17494)

Cherry-picked feat: add path & method labels to prometheus metrics for
current requests (#17362)

Closes: #17212

Co-authored-by: Michael Suchacz <203725896+ibetitsmike@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants