Skip to content

chore: add prometheus monitoring of workspace traffic generation #7583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
May 26, 2023
Merged

chore: add prometheus monitoring of workspace traffic generation #7583

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
f62cf16
add prometheus metrics endpoint to workspacetraffic command
johnstcn May 15, 2023
00b9eca
use self-managed prometheus
johnstcn May 16, 2023
9a9778c
plumb in prom properly
johnstcn May 16, 2023
f00b8d7
fix label on second pod monitor
johnstcn May 16, 2023
69bc5ad
do not clobber existing prometheus
johnstcn May 16, 2023
57d338d
remote_write all metrics
johnstcn May 17, 2023
a1d14df
add convenience script to create traffic inside cluster
johnstcn May 17, 2023
fb96c4d
fixup! add convenience script to create traffic inside cluster
johnstcn May 17, 2023
59ef445
make fmt, make gen
johnstcn May 17, 2023
6734521
Update scaletest/terraform/prometheus.tf
johnstcn May 17, 2023
4ab2314
Merge remote-tracking branch 'origin/main' into cj/workspacetraffic-p…
johnstcn May 24, 2023
d54af33
address PR comments
johnstcn May 24, 2023
8c48d2b
move test to workspacetraffic package
johnstcn May 25, 2023
cf00b5a
Merge remote-tracking branch 'origin/main' into cj/workspacetraffic-p…
johnstcn May 25, 2023
e6917e6
update golden files
johnstcn May 25, 2023
ea71c4f
separate prom address var
johnstcn May 25, 2023
399e506
break out errors in to read/write
johnstcn May 26, 2023
ec96a00
ignore certain errors such as websocket closing or context cancellation
johnstcn May 26, 2023
a4ecd0c
test metrics more closely
johnstcn May 26, 2023
fe0ecfc
fixup! test metrics more closely
johnstcn May 26, 2023
239ba96
add wait for prometheus metrics
johnstcn May 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,3 +59,4 @@ site/stats/
./scaletest/terraform/.terraform
./scaletest/terraform/.terraform.lock.hcl
terraform.tfstate.*
**/*.tfvars
1 change: 1 addition & 0 deletions .prettierignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ site/stats/
./scaletest/terraform/.terraform
./scaletest/terraform/.terraform.lock.hcl
terraform.tfstate.*
**/*.tfvars
# .prettierignore.include:
# Helm templates contain variables that are invalid YAML and can't be formatted
# by Prettier.
Expand Down
55 changes: 40 additions & 15 deletions cli/scaletest.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,14 @@ import (
"time"

"github.com/google/uuid"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"go.opentelemetry.io/otel/trace"
"golang.org/x/xerrors"

"cdr.dev/slog"
"cdr.dev/slog/sloggers/sloghuman"

"github.com/coder/coder/cli/clibase"
"github.com/coder/coder/cli/cliui"
"github.com/coder/coder/coderd/httpapi"
Expand Down Expand Up @@ -896,13 +901,14 @@ func (r *RootCmd) scaletestCreateWorkspaces() *clibase.Cmd {

func (r *RootCmd) scaletestWorkspaceTraffic() *clibase.Cmd {
var (
tickInterval time.Duration
bytesPerTick int64
client = &codersdk.Client{}
tracingFlags = &scaletestTracingFlags{}
strategy = &scaletestStrategyFlags{}
cleanupStrategy = &scaletestStrategyFlags{cleanup: true}
output = &scaletestOutputFlags{}
tickInterval time.Duration
bytesPerTick int64
scaletestPrometheusAddress string
client = &codersdk.Client{}
tracingFlags = &scaletestTracingFlags{}
strategy = &scaletestStrategyFlags{}
cleanupStrategy = &scaletestStrategyFlags{cleanup: true}
output = &scaletestOutputFlags{}
)

cmd := &clibase.Cmd{
Expand All @@ -913,6 +919,12 @@ func (r *RootCmd) scaletestWorkspaceTraffic() *clibase.Cmd {
),
Handler: func(inv *clibase.Invocation) error {
ctx := inv.Context()
reg := prometheus.NewRegistry()
metrics := workspacetraffic.NewMetrics(reg, "username", "workspace_name", "agent_name")

logger := slog.Make(sloghuman.Sink(io.Discard))
prometheusSrvClose := ServeHandler(ctx, logger, promhttp.HandlerFor(reg, promhttp.HandlerOpts{}), scaletestPrometheusAddress, "prometheus")
defer prometheusSrvClose()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we need to add some graceful period before closing to make sure that all relevant metrics are scraped before the tool goes down.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this would be heavily dependent on the prometheus scrape interval. Simplest is probably to expose it as a parameter to be set by the test operatorl.


// Bypass rate limiting
client.HTTPClient = &http.Client{
Expand Down Expand Up @@ -955,16 +967,18 @@ func (r *RootCmd) scaletestWorkspaceTraffic() *clibase.Cmd {
th := harness.NewTestHarness(strategy.toStrategy(), cleanupStrategy.toStrategy())
for idx, ws := range workspaces {
var (
agentID uuid.UUID
name = "workspace-traffic"
id = strconv.Itoa(idx)
agentID uuid.UUID
agentName string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't go through the whole PR, but why is it required to specify the agent name? isn't it always "main"?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that the agent name maps directly to what's specified in the workspace Terraform, is this not the case?

name = "workspace-traffic"
id = strconv.Itoa(idx)
)

for _, res := range ws.LatestBuild.Resources {
if len(res.Agents) == 0 {
continue
}
agentID = res.Agents[0].ID
agentName = res.Agents[0].Name
}

if agentID == uuid.Nil {
Expand All @@ -974,16 +988,20 @@ func (r *RootCmd) scaletestWorkspaceTraffic() *clibase.Cmd {

// Setup our workspace agent connection.
config := workspacetraffic.Config{
AgentID: agentID,
BytesPerTick: bytesPerTick,
Duration: strategy.timeout,
TickInterval: tickInterval,
AgentID: agentID,
AgentName: agentName,
BytesPerTick: bytesPerTick,
Duration: strategy.timeout,
TickInterval: tickInterval,
WorkspaceName: ws.Name,
WorkspaceOwner: ws.OwnerName,
Registry: reg,
}

if err := config.Validate(); err != nil {
return xerrors.Errorf("validate config: %w", err)
}
var runner harness.Runnable = workspacetraffic.NewRunner(client, config)
var runner harness.Runnable = workspacetraffic.NewRunner(client, config, metrics)
if tracingEnabled {
runner = &runnableTraceWrapper{
tracer: tracer,
Expand Down Expand Up @@ -1034,6 +1052,13 @@ func (r *RootCmd) scaletestWorkspaceTraffic() *clibase.Cmd {
Description: "How often to send traffic.",
Value: clibase.DurationOf(&tickInterval),
},
{
Flag: "scaletest-prometheus-address",
Env: "CODER_SCALETEST_PROMETHEUS_ADDRESS",
Default: "0.0.0.0:21112",
Description: "Address on which to expose scaletest prometheus metrics.",
Value: clibase.StringOf(&scaletestPrometheusAddress),
},
}

tracingFlags.attach(&cmd.Options)
Expand Down
3 changes: 3 additions & 0 deletions cli/testdata/coder_scaletest_workspace-traffic_--help.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ Generate traffic to scaletest workspaces through coderd
Output format specs in the format "<format>[:<path>]". Not specifying
a path will default to stdout. Available formats: text, json.

--scaletest-prometheus-address string, $CODER_SCALETEST_PROMETHEUS_ADDRESS (default: 0.0.0.0:21112)
Address on which to expose scaletest prometheus metrics.

--tick-interval duration, $CODER_SCALETEST_WORKSPACE_TRAFFIC_TICK_INTERVAL (default: 100ms)
How often to send traffic.

Expand Down
10 changes: 10 additions & 0 deletions docs/cli/scaletest_workspace-traffic.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,16 @@ Timeout per job. Jobs may take longer to complete under higher concurrency limit

Output format specs in the format "<format>[:<path>]". Not specifying a path will default to stdout. Available formats: text, json.

### --scaletest-prometheus-address

| | |
| ----------- | ------------------------------------------------ |
| Type | <code>string</code> |
| Environment | <code>$CODER_SCALETEST_PROMETHEUS_ADDRESS</code> |
| Default | <code>0.0.0.0:21112</code> |

Address on which to expose scaletest prometheus metrics.

### --tick-interval

| | |
Expand Down
7 changes: 5 additions & 2 deletions scaletest/terraform/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,12 @@ project_id = "some_google_project_id"
1. Run `coder_init.sh <coder_url>` to setup an initial user and a pre-configured Kubernetes
template. It will also download the Coder CLI from the Coder instance locally.

1. Do whatever you need to do with the Coder instance.
1. Do whatever you need to do with the Coder instance:

> To run Coder commands against the instance, you can use `coder_shim.sh <command>`.
> Note: To run Coder commands against the instance, you can use `coder_shim.sh <command>`.
> You don't need to run `coder login` yourself.

- To create workspaces, run `./coder_shim.sh scaletest create-workspaces --template="kubernetes" --count=N`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that at some point you may want to include also rich parameters as they cause more DB calls in total (worst case scenario).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that would be a good test for load related to concurrent workspace builds.
For the moment though, I'm going to keep it simple with zero parameters.

- To generate workspace traffic, run `./coder_trafficgen.sh <name of loadtest from your Terraform vars>`. This will keep running until you delete the pod `coder-scaletest-workspace-traffic`.

1. When you are finished, you can run `terraform destroy -var-file=override.tfvars`.
28 changes: 0 additions & 28 deletions scaletest/terraform/coder.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,34 +128,6 @@ EOF
]
}

resource "local_file" "coder-monitoring-manifest" {
filename = "${path.module}/.coderv2/coder-monitoring.yaml"
content = <<EOF
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: ${kubernetes_namespace.coder_namespace.metadata.0.name}
name: coder-monitoring
spec:
selector:
matchLabels:
app.kubernetes.io/name: coder
endpoints:
- port: prometheus-http
interval: 30s
EOF
}

resource "null_resource" "coder-monitoring-manifest_apply" {
provisioner "local-exec" {
working_dir = "${abspath(path.module)}/.coderv2"
command = <<EOF
KUBECONFIG=${var.name}-cluster.kubeconfig gcloud container clusters get-credentials ${var.name}-cluster --project=${var.project_id} --zone=${var.zone} && \
KUBECONFIG=${var.name}-cluster.kubeconfig kubectl apply -f ${abspath(local_file.coder-monitoring-manifest.filename)}
EOF
}
}

resource "local_file" "kubernetes_template" {
filename = "${path.module}/.coderv2/templates/kubernetes/main.tf"
content = <<EOF
Expand Down
73 changes: 73 additions & 0 deletions scaletest/terraform/coder_workspacetraffic.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/usr/bin/env bash

set -euo pipefail

if [[ $# -lt 1 ]]; then
echo "Usage: $0 <loadtest name>"
exit 1
fi

# Allow toggling verbose output
[[ -n ${VERBOSE:-} ]] && set -x

LOADTEST_NAME="$1"
CODER_TOKEN=$(./coder_shim.sh tokens create)
CODER_URL="http://coder.coder-${LOADTEST_NAME}.svc.cluster.local"
export KUBECONFIG="${PWD}/.coderv2/${LOADTEST_NAME}-cluster.kubeconfig"

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: coder-scaletest-workspace-traffic
namespace: coder-${LOADTEST_NAME}
labels:
app.kubernetes.io/name: coder-scaletest-workspace-traffic
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.google.com/gke-nodepool
operator: In
values:
- ${LOADTEST_NAME}-misc
containers:
- command:
- sh
- -c
- "curl -fsSL $CODER_URL/bin/coder-linux-amd64 -o /tmp/coder && chmod +x /tmp/coder && /tmp/coder --url=$CODER_URL --token=$CODER_TOKEN scaletest workspace-traffic"
env:
- name: CODER_URL
value: $CODER_URL
- name: CODER_TOKEN
value: $CODER_TOKEN
- name: CODER_SCALETEST_PROMETHEUS_ADDRESS
value: "0.0.0.0:21112"
- name: CODER_SCALETEST_JOB_TIMEOUT
value: "30m"
- name: CODER_SCALETEST_CONCURRENCY
value: "0"
- name: CODER_SCALETEST_WORKSPACE_TRAFFIC_BYTES_PER_TICK
value: "2048"
ports:
- containerPort: 21112
name: prometheus-http
protocol: TCP
name: cli
image: docker.io/codercom/enterprise-minimal:ubuntu
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
namespace: coder-${LOADTEST_NAME}
name: coder-workspacetraffic-monitoring
spec:
selector:
matchLabels:
app.kubernetes.io/name: coder-scaletest-workspace-traffic
podMetricsEndpoints:
- port: prometheus-http
interval: 15s
EOF
2 changes: 1 addition & 1 deletion scaletest/terraform/gcp_cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ resource "google_container_cluster" "primary" {
monitoring_config {
enable_components = ["SYSTEM_COMPONENTS"]
managed_prometheus {
enabled = true
enabled = false
}
}
workload_identity_config {
Expand Down
Loading