Skip to content

feat: expose agent stats via Prometheus endpoint #7115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 43 commits into from
Apr 14, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
8d4e67d
WIP
mtojek Apr 3, 2023
da729e6
Merge branch 'main' into 6724-metrics
mtojek Apr 4, 2023
9ad09b2
WIP
mtojek Apr 4, 2023
440657c
WIP
mtojek Apr 5, 2023
8764f89
Agents
mtojek Apr 5, 2023
663b5d5
fix
mtojek Apr 5, 2023
63aff5e
1min
mtojek Apr 5, 2023
3905481
fix
mtojek Apr 5, 2023
f8d6f46
WIP
mtojek Apr 5, 2023
d487a77
Test
mtojek Apr 5, 2023
7acbaf0
docs
mtojek Apr 5, 2023
7418779
fmt
mtojek Apr 5, 2023
3a8e4e6
Add timer to measure the metrics collection
mtojek Apr 6, 2023
b5d0581
Use CachedGaugeVec
mtojek Apr 6, 2023
e4d708b
Unit tests
mtojek Apr 6, 2023
199e549
WIP
mtojek Apr 7, 2023
7307bd3
Merge branch 'main' into 6724-metrics-2
mtojek Apr 12, 2023
d0b8398
WIP
mtojek Apr 13, 2023
f0c0418
db: GetWorkspaceAgentStatsAndLabels
mtojek Apr 13, 2023
970d35a
fmt
mtojek Apr 13, 2023
229f546
WIP
mtojek Apr 13, 2023
7070e0e
Merge branch 'main' into 6724-metrics-2
mtojek Apr 13, 2023
8c6f96b
gauges
mtojek Apr 13, 2023
1ed37b4
feat: collect
mtojek Apr 13, 2023
7ee1bfc
fix
mtojek Apr 13, 2023
2b8a9e4
fmt
mtojek Apr 13, 2023
322f7e8
minor fixes
mtojek Apr 14, 2023
c7af75a
Prometheus flag
mtojek Apr 14, 2023
9693fa8
fix
mtojek Apr 14, 2023
28f7a13
WIP
mtojek Apr 14, 2023
7878167
fix tests
mtojek Apr 14, 2023
d9e4903
WIP
mtojek Apr 14, 2023
0d37c85
fix json
mtojek Apr 14, 2023
f752c6f
Rx Tx bytes
mtojek Apr 14, 2023
9c7aef8
CloseFunc
mtojek Apr 14, 2023
5290571
fix
mtojek Apr 14, 2023
1cbe59b
fix
mtojek Apr 14, 2023
f8f11eb
Fixes
mtojek Apr 14, 2023
4ffae11
fix
mtojek Apr 14, 2023
7ba16b5
fix: IgnoreErrors
mtojek Apr 14, 2023
2a4c674
Fix: Windows
mtojek Apr 14, 2023
201da83
fix
mtojek Apr 14, 2023
ba52c45
reflect.DeepEquals
mtojek Apr 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Prometheus flag
  • Loading branch information
mtojek committed Apr 14, 2023
commit c7af75a318f0b152d3f24a03900e560ac0fee1d6
10 changes: 6 additions & 4 deletions cli/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -844,11 +844,13 @@ func (r *RootCmd) Server(newAPI func(context.Context, *coderd.Options) (*coderd.
}
defer closeWorkspacesFunc()

closeAgentStatsFunc, err := prometheusmetrics.AgentStats(ctx, logger, options.PrometheusRegistry, options.Database, time.Now(), 0)
if err != nil {
return xerrors.Errorf("register agent stats prometheus metric: %w", err)
if cfg.Prometheus.CollectAgentStats {
closeAgentStatsFunc, err := prometheusmetrics.AgentStats(ctx, logger, options.PrometheusRegistry, options.Database, time.Now(), 0)
if err != nil {
return xerrors.Errorf("register agent stats prometheus metric: %w", err)
}
defer closeAgentStatsFunc()
}
defer closeAgentStatsFunc()

//nolint:revive
defer serveHandler(ctx, logger, promhttp.InstrumentMetricHandler(
Expand Down
3 changes: 3 additions & 0 deletions cli/testdata/coder_server_--help.golden
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,9 @@ Use a YAML configuration file when your server launch become unwieldy.
--prometheus-address host:port, $CODER_PROMETHEUS_ADDRESS (default: 127.0.0.1:2112)
The bind address to serve prometheus metrics.

--prometheus-collect-agent-stats bool, $CODER_PROMETHEUS_COLLECT_AGENT_STATS
Collect agent stats (may increase charges for metrics storage) .

--prometheus-enable bool, $CODER_PROMETHEUS_ENABLE
Serve prometheus metrics on the address defined by prometheus address.

Expand Down
3 changes: 3 additions & 0 deletions cli/testdata/server-config.yaml.golden
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,9 @@ introspection:
# The bind address to serve prometheus metrics.
# (default: 127.0.0.1:2112, type: host:port)
address: 127.0.0.1:2112
# Collect agent stats (may increase charges for metrics storage) .
# (default: <unset>, type: bool)
collect_agent_stats: false
pprof:
# Serve pprof metrics on the address defined by pprof address.
# (default: <unset>, type: bool)
Expand Down
3 changes: 3 additions & 0 deletions coderd/apidoc/docs.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions coderd/apidoc/swagger.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 12 additions & 2 deletions codersdk/deployment.go
Original file line number Diff line number Diff line change
Expand Up @@ -225,8 +225,9 @@ type DERPConfig struct {
}

type PrometheusConfig struct {
Enable clibase.Bool `json:"enable" typescript:",notnull"`
Address clibase.HostPort `json:"address" typescript:",notnull"`
Enable clibase.Bool `json:"enable" typescript:",notnull"`
Address clibase.HostPort `json:"address" typescript:",notnull"`
CollectAgentStats clibase.Bool `json:"collect_agent_stats" typescript:",notnull"`
}

type PprofConfig struct {
Expand Down Expand Up @@ -722,6 +723,15 @@ when required by your organization's security policy.`,
Group: &deploymentGroupIntrospectionPrometheus,
YAML: "address",
},
{
Name: "Prometheus Collect Agent Stats",
Description: "Collect agent stats (may increase charges for metrics storage) .",
Flag: "prometheus-collect-agent-stats",
Env: "CODER_PROMETHEUS_COLLECT_AGENT_STATS",
Value: &c.Prometheus.CollectAgentStats,
Group: &deploymentGroupIntrospectionPrometheus,
YAML: "collect_agent_stats",
},
// Pprof settings
{
Name: "pprof Enable",
Expand Down
1 change: 1 addition & 0 deletions docs/api/general.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,7 @@ curl -X GET http://coder-server:8080/api/v2/deployment/config \
"host": "string",
"port": "string"
},
"collect_agent_stats": true,
"enable": true
},
"provisioner": {
Expand Down
12 changes: 8 additions & 4 deletions docs/api/schemas.md
Original file line number Diff line number Diff line change
Expand Up @@ -1901,6 +1901,7 @@ CreateParameterRequest is a structure used to create a new parameter value for a
"host": "string",
"port": "string"
},
"collect_agent_stats": true,
"enable": true
},
"provisioner": {
Expand Down Expand Up @@ -2244,6 +2245,7 @@ CreateParameterRequest is a structure used to create a new parameter value for a
"host": "string",
"port": "string"
},
"collect_agent_stats": true,
"enable": true
},
"provisioner": {
Expand Down Expand Up @@ -3155,16 +3157,18 @@ Parameter represents a set value for the scope.
"host": "string",
"port": "string"
},
"collect_agent_stats": true,
"enable": true
}
```

### Properties

| Name | Type | Required | Restrictions | Description |
| --------- | ------------------------------------ | -------- | ------------ | ----------- |
| `address` | [clibase.HostPort](#clibasehostport) | false | | |
| `enable` | boolean | false | | |
| Name | Type | Required | Restrictions | Description |
| --------------------- | ------------------------------------ | -------- | ------------ | ----------- |
| `address` | [clibase.HostPort](#clibasehostport) | false | | |
| `collect_agent_stats` | boolean | false | | |
| `enable` | boolean | false | | |

## codersdk.ProvisionerConfig

Expand Down
10 changes: 10 additions & 0 deletions docs/cli/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -555,6 +555,16 @@ URL of a PostgreSQL database. If empty, PostgreSQL binaries will be downloaded f

The bind address to serve prometheus metrics.

### --prometheus-collect-agent-stats

| | |
| ----------- | --------------------------------------------------------- |
| Type | <code>bool</code> |
| Environment | <code>$CODER_PROMETHEUS_COLLECT_AGENT_STATS</code> |
| YAML | <code>introspection.prometheus.collect_agent_stats</code> |

Collect agent stats (may increase charges for metrics storage) .

### --prometheus-enable

| | |
Expand Down
1 change: 1 addition & 0 deletions site/src/api/typesGenerated.ts
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,7 @@ export interface PrometheusConfig {
// Named type "github.com/coder/coder/cli/clibase.HostPort" unknown, using "any"
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- External type
readonly address: any
readonly collect_agent_stats: boolean
}

// From codersdk/deployment.go
Expand Down