Skip to content

chore: add prometheus monitoring of workspace traffic generation #7583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
May 26, 2023
Merged

chore: add prometheus monitoring of workspace traffic generation #7583

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
f62cf16
add prometheus metrics endpoint to workspacetraffic command
johnstcn May 15, 2023
00b9eca
use self-managed prometheus
johnstcn May 16, 2023
9a9778c
plumb in prom properly
johnstcn May 16, 2023
f00b8d7
fix label on second pod monitor
johnstcn May 16, 2023
69bc5ad
do not clobber existing prometheus
johnstcn May 16, 2023
57d338d
remote_write all metrics
johnstcn May 17, 2023
a1d14df
add convenience script to create traffic inside cluster
johnstcn May 17, 2023
fb96c4d
fixup! add convenience script to create traffic inside cluster
johnstcn May 17, 2023
59ef445
make fmt, make gen
johnstcn May 17, 2023
6734521
Update scaletest/terraform/prometheus.tf
johnstcn May 17, 2023
4ab2314
Merge remote-tracking branch 'origin/main' into cj/workspacetraffic-p…
johnstcn May 24, 2023
d54af33
address PR comments
johnstcn May 24, 2023
8c48d2b
move test to workspacetraffic package
johnstcn May 25, 2023
cf00b5a
Merge remote-tracking branch 'origin/main' into cj/workspacetraffic-p…
johnstcn May 25, 2023
e6917e6
update golden files
johnstcn May 25, 2023
ea71c4f
separate prom address var
johnstcn May 25, 2023
399e506
break out errors in to read/write
johnstcn May 26, 2023
ec96a00
ignore certain errors such as websocket closing or context cancellation
johnstcn May 26, 2023
a4ecd0c
test metrics more closely
johnstcn May 26, 2023
fe0ecfc
fixup! test metrics more closely
johnstcn May 26, 2023
239ba96
add wait for prometheus metrics
johnstcn May 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address PR comments
  • Loading branch information
@johnstcn
johnstcn committed May 24, 2023
commit d54af33c10b0747bf38e319f8f86281d87dc9c65
2 changes: 1 addition & 1 deletion cli/scaletest.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1056,7 +1056,7 @@ func (r *RootCmd) scaletestWorkspaceTraffic() *clibase.Cmd {
Flag: "scaletest-prometheus-address",
Env: "CODER_SCALETEST_PROMETHEUS_ADDRESS",
Default: "0.0.0.0:21112",
Description: "Address on which to expose scaletest prometheus metrics.",
Description: "Address on which to expose scaletest Prometheus metrics.",
Value: clibase.StringOf(&scaletestPrometheusAddress),
},
}
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/scaletest_workspace-traffic.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Output format specs in the format "<format>[:<path>]". Not specifying a path wil
| Environment | <code>$CODER_SCALETEST_PROMETHEUS_ADDRESS</code> |
| Default | <code>0.0.0.0:21112</code> |

Address on which to expose scaletest prometheus metrics.
Address on which to expose scaletest Prometheus metrics.

### --tick-interval

Expand Down
20 changes: 10 additions & 10 deletions scaletest/workspacetraffic/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ package workspacetraffic
import "github.com/prometheus/client_golang/prometheus"

type Metrics struct {
BytesRead prometheus.CounterVec
BytesWritten prometheus.CounterVec
Errors prometheus.CounterVec
ReadLatencyMS prometheus.HistogramVec
WriteLatencyMS prometheus.HistogramVec
LabelNames []string
BytesRead prometheus.CounterVec
BytesWritten prometheus.CounterVec
Errors prometheus.CounterVec
ReadLatencySeconds prometheus.HistogramVec
WriteLatencySeconds prometheus.HistogramVec
LabelNames []string
}

func NewMetrics(reg prometheus.Registerer, labelNames ...string) *Metrics {
Expand All @@ -28,12 +28,12 @@ func NewMetrics(reg prometheus.Registerer, labelNames ...string) *Metrics {
Subsystem: "scaletest",
Name: "errors",
}, labelNames),
ReadLatencyMS: *prometheus.NewHistogramVec(prometheus.HistogramOpts{
ReadLatencySeconds: *prometheus.NewHistogramVec(prometheus.HistogramOpts{
Namespace: "coderd",
Subsystem: "scaletest",
Name: "read_latency_seconds",
}, labelNames),
WriteLatencyMS: *prometheus.NewHistogramVec(prometheus.HistogramOpts{
WriteLatencySeconds: *prometheus.NewHistogramVec(prometheus.HistogramOpts{
Namespace: "coderd",
Subsystem: "scaletest",
Name: "write_latency_seconds",
Expand All @@ -43,7 +43,7 @@ func NewMetrics(reg prometheus.Registerer, labelNames ...string) *Metrics {
reg.MustRegister(m.BytesRead)
reg.MustRegister(m.BytesWritten)
reg.MustRegister(m.Errors)
reg.MustRegister(m.ReadLatencyMS)
reg.MustRegister(m.WriteLatencyMS)
reg.MustRegister(m.ReadLatencySeconds)
reg.MustRegister(m.WriteLatencySeconds)
return m
}
4 changes: 2 additions & 2 deletions scaletest/workspacetraffic/run.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ type countReadWriter struct {
func (w *countReadWriter) Read(p []byte) (int, error) {
start := time.Now()
n, err := w.ReadWriter.Read(p)
w.metrics.ReadLatencyMS.WithLabelValues(w.labels...).Observe(time.Since(start).Seconds())
w.metrics.ReadLatencySeconds.WithLabelValues(w.labels...).Observe(time.Since(start).Seconds())
if n > 0 {
w.bytesRead.Add(int64(n))
w.metrics.BytesRead.WithLabelValues(w.labels...).Add(float64(n))
Expand All @@ -206,7 +206,7 @@ func (w *countReadWriter) Read(p []byte) (int, error) {
func (w *countReadWriter) Write(p []byte) (int, error) {
start := time.Now()
n, err := w.ReadWriter.Write(p)
w.metrics.WriteLatencyMS.WithLabelValues(w.labels...).Observe(time.Since(start).Seconds())
w.metrics.WriteLatencySeconds.WithLabelValues(w.labels...).Observe(time.Since(start).Seconds())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious if we need another metric (bytes?) to calculate the bandwidth. As far as I see, there are only WriteLatencySeconds and total BytesWritten. I'm curious if we need a metric to calculate "bytes per write in time"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that bytes per tick and ticks per second are inputs to this, I think we should probably be OK. But good food for thought!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You calc throughput by the (count at time X) - (count at time Y). We should not be doing that math in coderd imo

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, will address in a follow-up

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prometheus itself will happily compute the time derivative of any metric, so a counter of bytes written is all we need.

if n > 0 {
w.bytesWritten.Add(int64(n))
w.metrics.BytesWritten.WithLabelValues(w.labels...).Add(float64(n))
Expand Down