Skip to content

docs: Prometheus metrics + generator #5179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Nov 30, 2022
Merged

Conversation

mtojek
Copy link
Member

@mtojek mtojek commented Nov 28, 2022

Fixes: #3520

Preview: https://github.com/mtojek/coder/blob/3520-doc-prom-metrics/docs/admin/prometheus.md

This PR adds another doc page, that describes the Prometheus endpoint. I didn't list or describe all metrics on purpose, so we won't end up with outdated content. I added the doc generator, which converts the Prometheus dump into the Markdown table.

@mtojek mtojek self-assigned this Nov 28, 2022
@mtojek mtojek requested a review from a team November 28, 2022 17:04
@mtojek mtojek marked this pull request as ready for review November 28, 2022 17:14
mtojek and others added 6 commits November 28, 2022 18:32
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
@mtojek mtojek mentioned this pull request Nov 28, 2022
3 tasks
@mtojek mtojek marked this pull request as draft November 29, 2022 07:56
@mtojek mtojek marked this pull request as ready for review November 29, 2022 14:30
@mtojek mtojek requested a review from a team November 29, 2022 15:00
@mtojek mtojek changed the title docs: Prometheus metrics docs: Prometheus metrics + generator Nov 29, 2022
coderd_api_concurrent_websockets 1
# HELP coderd_api_request_latencies_ms Latency distribution of requests in milliseconds
# TYPE coderd_api_request_latencies_ms histogram
coderd_api_request_latencies_ms_bucket{method="GET",path="",le="1"} 10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if we could get a cleaner dump here, the histogram seems to take up a lot of space, but it's just repeating different routes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frankly speaking, the idea was to just dump it and forget about it. The format is not intended to be reviewed by humans. The generator makes sure that it can be cleaned and processed into the Markdown table.

Regarding the dumping process, I considered other options like booting up the coderd and fetching Prometheus metrics from http://localhost:2112, but you would have to run it in a container as process metrics are only available in Linux. Eventually, I decided to leave it to the developer to just use curl and update the file.

If you have any ideas on how to improve it, I'm happy to change the concept.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if instead of storing the dump in the repo you made the script directly connect to the server and download the prom metrics? Would probably need to be moved out of make gen as it would be prone to failure if not configured properly.

Copy link
Member Author

@mtojek mtojek Nov 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said before, in this case, the coder server must be started in the container, so it can also run the process stats collector (Linux only). I'm looking into fully reproducible options.

Most likely, the script would have to simulate some actions, start the workspace, provision something, etc. to collect as many metrics as possible.

Long story short, this method isn't perfect too.

EDIT:

Unless you're thinking about detaching the script from make gen totally, but I guess that the idea of having a container might be worth considering here as well.

@mtojek
Copy link
Member Author

mtojek commented Nov 30, 2022

I've just re-requested reviews. Kindly please to take another look and if it's ok, please 👍 .

@mtojek mtojek merged commit 38bdae7 into coder:main Nov 30, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Nov 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

docs: observability & prometheus
5 participants