diff --git a/admin/prometheus.md b/admin/prometheus.md index ef8761835..6d68359a2 100644 --- a/admin/prometheus.md +++ b/admin/prometheus.md @@ -58,8 +58,8 @@ deleted. ### Drop workspace metrics config [Prometheus Documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs) -about relabelling metrics. In this case we will drop all metrics that contain the -`workspace_id` label. +about relabelling metrics. In this case we will drop all metrics that contain +the `workspace_id` label. ```yaml metric_relabel_configs: @@ -94,6 +94,12 @@ Below is a list of the various metrics emitted by Coder's Prometheus endpoint: | Metric | Type | Description | | ----------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `coderd_agent_aggregator_agent_push_backlog` | `gauge` | Total number of agent metric bundles waiting to be processed. | +| `coderd_agent_aggregator_collect_backlog` | `gauge` | Total amount of gathers waiting to collect metrics. | +| `coderd_agent_aggregator_collect_nanoseconds` | `summary` | Time taken to collect all metrics. | +| `coderd_agent_aggregator_count_total` | `gauge` | Total number of agent metrics being reported by this coderd. | +| `coderd_agent_aggregator_delete_backlog` | `gauge` | Total number of agents waiting to be deleted in aggregator. | +| `coderd_agent_aggregator_workspace_count_total` | `gauge` | Total number of workspace agents pushing metrics to this coderd. | | `coderd_api_concurrent_requests` | `gauge` | The total number of concurrent API requests | | `coderd_api_concurrent_websockets` | `gauge` | The total number of concurrent API websockets | | `coderd_api_request_latencies_ms` | `histogram` | Latency distribution of requests in milliseconds | @@ -123,16 +129,16 @@ Below is a list of the various metrics emitted by Coder's Prometheus endpoint: | `go_gc_cycles_forced_gc_cycles_total` | `counter` | Count of completed GC cycles forced by the application. | | `go_gc_cycles_total_gc_cycles_total` | `counter` | Count of all completed GC cycles. | | `go_gc_duration_seconds` | `summary` | A summary of the pause duration of garbage collection cycles. | -| `go_gc_heap_allocs_by_size_bytes_total` | `histogram` | Distribution of heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. | +| `go_gc_heap_allocs_by_size_bytes` | `histogram` | Distribution of heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. | | `go_gc_heap_allocs_bytes_total` | `counter` | Cumulative sum of memory allocated to the heap by the application. | | `go_gc_heap_allocs_objects_total` | `counter` | Cumulative count of heap allocations triggered by the application. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. | -| `go_gc_heap_frees_by_size_bytes_total` | `histogram` | Distribution of freed heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. | +| `go_gc_heap_frees_by_size_bytes` | `histogram` | Distribution of freed heap allocations by approximate size. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. | | `go_gc_heap_frees_bytes_total` | `counter` | Cumulative sum of heap memory freed by the garbage collector. | | `go_gc_heap_frees_objects_total` | `counter` | Cumulative count of heap allocations whose storage was freed by the garbage collector. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. | | `go_gc_heap_goal_bytes` | `gauge` | Heap size target for the end of the GC cycle. | | `go_gc_heap_objects_objects` | `gauge` | Number of objects, live or unswept, occupying heap memory. | | `go_gc_heap_tiny_allocs_objects_total` | `counter` | Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. | -| `go_gc_pauses_seconds_total` | `histogram` | Distribution individual GC-related stop-the-world pause latencies. | +| `go_gc_pauses_seconds` | `histogram` | Distribution individual GC-related stop-the-world pause latencies. | | `go_goroutines` | `gauge` | Number of goroutines that currently exist. | | `go_info` | `gauge` | Information about the Go environment. | | `go_memory_classes_heap_free_bytes` | `gauge` | Memory that is completely free and eligible to be returned to the underlying system, but has not been. This metric is the runtime's estimate of free address space that is backed by physical memory. | @@ -149,11 +155,10 @@ Below is a list of the various metrics emitted by Coder's Prometheus endpoint: | `go_memory_classes_other_bytes` | `gauge` | Memory used by execution trace buffers, structures for debugging the runtime, finalizer and profiler specials, and more. | | `go_memory_classes_profiling_buckets_bytes` | `gauge` | Memory that is used by the stack trace hash map used for profiling. | | `go_memory_classes_total_bytes` | `gauge` | All memory mapped by the Go runtime into the current process as read-write. Note that this does not include memory mapped by code called via cgo or via the syscall package. Sum of all metrics in /memory/classes. | -| `go_memstats_alloc_bytes_total` | `counter` | Total number of bytes allocated, even if freed. | | `go_memstats_alloc_bytes` | `gauge` | Number of bytes allocated and still in use. | +| `go_memstats_alloc_bytes_total` | `counter` | Total number of bytes allocated, even if freed. | | `go_memstats_buck_hash_sys_bytes` | `gauge` | Number of bytes used by the profiling bucket hash table. | | `go_memstats_frees_total` | `counter` | Total number of frees. | -| `go_memstats_gc_cpu_fraction` | `gauge` | The fraction of this program's available CPU time used by the GC since the program started. | | `go_memstats_gc_sys_bytes` | `gauge` | Number of bytes used for garbage collection system metadata. | | `go_memstats_heap_alloc_bytes` | `gauge` | Number of heap bytes allocated and still in use. | | `go_memstats_heap_idle_bytes` | `gauge` | Number of heap bytes waiting to be used. | @@ -175,6 +180,15 @@ Below is a list of the various metrics emitted by Coder's Prometheus endpoint: | `go_memstats_sys_bytes` | `gauge` | Number of bytes obtained from system. | | `go_sched_goroutines_goroutines` | `gauge` | Count of live goroutines. | | `go_sched_latencies_seconds` | `histogram` | Distribution of the time goroutines have spent in the scheduler in a runnable state before actually running. | +| `go_sql_idle_connections` | `gauge` | The number of idle connections. | +| `go_sql_in_use_connections` | `gauge` | The number of connections currently in use. | +| `go_sql_max_idle_closed_total` | `counter` | The total number of connections closed due to SetMaxIdleConns. | +| `go_sql_max_idle_time_closed_total` | `counter` | The total number of connections closed due to SetConnMaxIdleTime. | +| `go_sql_max_lifetime_closed_total` | `counter` | The total number of connections closed due to SetConnMaxLifetime. | +| `go_sql_max_open_connections` | `gauge` | Maximum number of open connections to the database. | +| `go_sql_open_connections` | `gauge` | The number of established connections both in use and idle. | +| `go_sql_wait_count_total` | `counter` | The total number of connections waited for. | +| `go_sql_wait_duration_seconds_total` | `counter` | The total time blocked waiting for a new connection. | | `go_threads` | `gauge` | Number of OS threads created. | | `process_cpu_seconds_total` | `counter` | Total user and system CPU time spent in seconds. | | `process_max_fds` | `gauge` | Maximum number of open file descriptors. |