diff --git a/docs/platforms/kubernetes/coder-logstream-kube-logs-normal.png b/docs/platforms/kubernetes/coder-logstream-kube-logs-normal.png new file mode 100644 index 0000000000000..5947baca1919d Binary files /dev/null and b/docs/platforms/kubernetes/coder-logstream-kube-logs-normal.png differ diff --git a/docs/platforms/kubernetes/coder-logstream-kube-logs-pod-crashed.png b/docs/platforms/kubernetes/coder-logstream-kube-logs-pod-crashed.png new file mode 100644 index 0000000000000..4a5a5e183b6b5 Binary files /dev/null and b/docs/platforms/kubernetes/coder-logstream-kube-logs-pod-crashed.png differ diff --git a/docs/platforms/kubernetes/coder-logstream-kube-logs-quota-exceeded.png b/docs/platforms/kubernetes/coder-logstream-kube-logs-quota-exceeded.png new file mode 100644 index 0000000000000..f95506028d156 Binary files /dev/null and b/docs/platforms/kubernetes/coder-logstream-kube-logs-quota-exceeded.png differ diff --git a/docs/platforms/kubernetes/coder-logstream-kube-logs-wrong-image.png b/docs/platforms/kubernetes/coder-logstream-kube-logs-wrong-image.png new file mode 100644 index 0000000000000..5947baca1919d Binary files /dev/null and b/docs/platforms/kubernetes/coder-logstream-kube-logs-wrong-image.png differ diff --git a/docs/platforms/kubernetes/deployment-logs.md b/docs/platforms/kubernetes/deployment-logs.md new file mode 100644 index 0000000000000..d420cec49f74e --- /dev/null +++ b/docs/platforms/kubernetes/deployment-logs.md @@ -0,0 +1,59 @@ +# Kubernetes event logs + +To stream Kubernetes events into your workspace startup logs, you can use Coder's [`coder-logstream-kube`](https://github.com/coder/coder-logstream-kube) tool. `coder-logstream-kube` provides useful information about the workspace pod or deployment, such as: + +- Causes of pod provisioning failures, or why a pod is stuck in a pending state. +- Visibility into when pods are OOMKilled, or when they are evicted. + +## Prerequisites + +`coder-logstream-kube` works best with the [`kubernetes_deployment`](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/deployment) Terraform resource, which requires the `coder` service account to have permission to create deployments. For example, if you use [Helm](https://coder.com/docs/v2/latest/install/kubernetes#install-coder-with-helm) to install Coder, you should set `coder.serviceAccount.enableDeployments=true` in your `values.yaml` + +```diff +coder: +serviceAccount: + workspacePerms: true +- enableDeployments: false ++ enableDeployments: true + annotations: {} + name: coder +``` + +> Note: This is only required for Coder versions < 0.28.0, as this will be the default value for Coder versions >= 0.28.0 + +## Installation + +Install the `coder-kubestream-logs` helm chart on the cluster where the deployment is running. + +```shell +helm repo add coder-logstream-kube https://helm.coder.com/logstream-kube +helm install coder-logstream-kube coder-logstream-kube/coder-logstream-kube \ + --namespace coder \ + --set url= +``` + +## Example logs + +Here is an example of the logs you can expect to see in the workspace startup logs: + +### Normal pod deployment + +![normal pod deployment](./coder-logstream-kube-logs-normal.png) + +### Wrong image + +![Wrong image name](./coder-logstream-kube-logs-wrong-image.png) + +### Kubernetes quota exceeded + +![Kubernetes quota exceeded](./coder-logstream-kube-logs-quota-exceeded.png) + +### Pod crash loop + +![Pod crash loop](./coder-logstream-kube-logs-pod-crashed.png) + +## How it works + +Kubernetes provides an [informers](https://pkg.go.dev/k8s.io/client-go/informers) API that streams pod and event data from the API server. + +coder-logstream-kube listens for pod creation events with containers that have the CODER_AGENT_TOKEN environment variable set. All pod events are streamed as logs to the Coder API using the agent token for authentication. For more details, see the [coder-logstream-kube](https://github.com/coder/coder-logstream-kube) repository. diff --git a/docs/platforms/kubernetes/index.md b/docs/platforms/kubernetes/index.md index eef0e70e72c79..d52d10e4cddb8 100644 --- a/docs/platforms/kubernetes/index.md +++ b/docs/platforms/kubernetes/index.md @@ -18,7 +18,7 @@ From the dashboard, import the Kubernetes starter template: In the next screen, set the following template variables: -- use_kubeconfig: `false` (The ServiceAccount will authorize Coder to create pods on your cluster) +- `use_kubeconfig`: `false` (The ServiceAccount will authorize Coder to create pods on your cluster) - `namespace`: `coder` (or whatever namespace you deployed Coder on) ![Variables for Kubernetes template](../../images/platforms/kubernetes/template-variables.png) diff --git a/examples/templates/kubernetes/README.md b/examples/templates/kubernetes/README.md index 6c04af8ea6a63..01721b5caf215 100644 --- a/examples/templates/kubernetes/README.md +++ b/examples/templates/kubernetes/README.md @@ -7,7 +7,23 @@ icon: /icon/k8s.png # Getting started -This template creates a pod running the `codercom/enterprise-base:ubuntu` image. +This template creates a deplyment running the `codercom/enterprise-base:ubuntu` image. + +## Prerequisites + +This template uses [`kubernetes_deployment`](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/deployment) terraform resource, which requires the `coder` service account to have permission to create deploymnets. For example if you are using [helm](https://coder.com/docs/v2/latest/install/kubernetes#install-coder-with-helm) to install Coder, you should set `coder.serviceAccount.enableDeployments=true` in your `values.yaml` + +```diff +coder: +serviceAccount: + workspacePerms: true +- enableDeployments: false ++ enableDeployments: true + annotations: {} + name: coder +``` + +> Note: This is only required for Coder versions < 0.28.0, as this will be the default value for Coder versions >= 0.28.0 ## Authentication @@ -62,7 +78,7 @@ You may want to deploy workspaces on a cluster outside of the Coder control plan ## Namespace -The target namespace in which the pod will be deployed is defined via the `coder_workspace` +The target namespace in which the deployment will be deployed is defined via the `coder_workspace` variable. The namespace must exist prior to creating workspaces. ## Persistence @@ -96,3 +112,16 @@ resource "coder_agent" "main" { `code-server` is installed via the `startup_script` argument in the `coder_agent` resource block. The `coder_app` resource is defined to access `code-server` through the dashboard UI over `localhost:13337`. + +## Deployment logs + +To stream kubernetes pods events from the deployment, you can use Coder's [`coder-logstream-kube`](https://github.com/coder/coder-logstream-kube) tool. This can stream logs from the deployment to Coder's workspace startup logs. You just need to install the `coder-logstream-kube` helm chart on the cluster where the deployment is running. + +```shell +helm repo add coder-logstream-kube https://helm.coder.com/logstream-kube +helm install coder-logstream-kube coder-logstream-kube/coder-logstream-kube \ + --namespace coder \ + --set url= +``` + +For detailed instructions, see [Deployment logs](https://coder.com/docs/v2/latest/platforms/kubernetes/deployment-logs) diff --git a/examples/templates/kubernetes/main.tf b/examples/templates/kubernetes/main.tf index 5d414032382c9..7eef83dd45ae7 100644 --- a/examples/templates/kubernetes/main.tf +++ b/examples/templates/kubernetes/main.tf @@ -2,11 +2,11 @@ terraform { required_providers { coder = { source = "coder/coder" - version = "~> 0.7.0" + version = "~> 0.11.0" } kubernetes = { source = "hashicorp/kubernetes" - version = "~> 2.18" + version = "~> 2.22" } } } @@ -198,7 +198,7 @@ resource "kubernetes_persistent_volume_claim" "home" { "app.kubernetes.io/name" = "coder-pvc" "app.kubernetes.io/instance" = "coder-pvc-${lower(data.coder_workspace.me.owner)}-${lower(data.coder_workspace.me.name)}" "app.kubernetes.io/part-of" = "coder" - // Coder specific labels. + //Coder-specific labels. "com.coder.resource" = "true" "com.coder.workspace.id" = data.coder_workspace.me.id "com.coder.workspace.name" = data.coder_workspace.me.name @@ -220,8 +220,12 @@ resource "kubernetes_persistent_volume_claim" "home" { } } -resource "kubernetes_pod" "main" { +resource "kubernetes_deployment" "main" { count = data.coder_workspace.me.start_count + depends_on = [ + kubernetes_persistent_volume_claim.home + ] + wait_for_rollout = false metadata { name = "coder-${lower(data.coder_workspace.me.owner)}-${lower(data.coder_workspace.me.name)}" namespace = var.namespace @@ -229,73 +233,90 @@ resource "kubernetes_pod" "main" { "app.kubernetes.io/name" = "coder-workspace" "app.kubernetes.io/instance" = "coder-workspace-${lower(data.coder_workspace.me.owner)}-${lower(data.coder_workspace.me.name)}" "app.kubernetes.io/part-of" = "coder" - // Coder specific labels. - "com.coder.resource" = "true" - "com.coder.workspace.id" = data.coder_workspace.me.id - "com.coder.workspace.name" = data.coder_workspace.me.name - "com.coder.user.id" = data.coder_workspace.me.owner_id - "com.coder.user.username" = data.coder_workspace.me.owner + "com.coder.resource" = "true" + "com.coder.workspace.id" = data.coder_workspace.me.id + "com.coder.workspace.name" = data.coder_workspace.me.name + "com.coder.user.id" = data.coder_workspace.me.owner_id + "com.coder.user.username" = data.coder_workspace.me.owner } annotations = { "com.coder.user.email" = data.coder_workspace.me.owner_email } } + spec { - security_context { - run_as_user = "1000" - fs_group = "1000" - } - container { - name = "dev" - image = "codercom/enterprise-base:ubuntu" - image_pull_policy = "Always" - command = ["sh", "-c", coder_agent.main.init_script] - security_context { - run_as_user = "1000" - } - env { - name = "CODER_AGENT_TOKEN" - value = coder_agent.main.token - } - resources { - requests = { - "cpu" = "250m" - "memory" = "512Mi" - } - limits = { - "cpu" = "${data.coder_parameter.cpu.value}" - "memory" = "${data.coder_parameter.memory.value}Gi" - } - } - volume_mount { - mount_path = "/home/coder" - name = "home" - read_only = false + # replicas = data.coder_workspace.me.start_count + replicas = 1 + selector { + match_labels = { + "app.kubernetes.io/name" = "coder-workspace" } } - volume { - name = "home" - persistent_volume_claim { - claim_name = kubernetes_persistent_volume_claim.home.metadata.0.name - read_only = false + template { + metadata { + labels = { + "app.kubernetes.io/name" = "coder-workspace" + } } - } + spec { + security_context { + run_as_user = 1000 + fs_group = 1000 + } + + container { + name = "dev" + image = "codercom/enterprise-base:ubuntu" + image_pull_policy = "Always" + command = ["sh", "-c", coder_agent.main.init_script] + security_context { + run_as_user = "1000" + } + env { + name = "CODER_AGENT_TOKEN" + value = coder_agent.main.token + } + resources { + requests = { + "cpu" = "250m" + "memory" = "512Mi" + } + limits = { + "cpu" = "${data.coder_parameter.cpu.value}" + "memory" = "${data.coder_parameter.memory.value}Gi" + } + } + volume_mount { + mount_path = "/home/coder" + name = "home" + read_only = false + } + } + volume { + name = "home" + persistent_volume_claim { + claim_name = kubernetes_persistent_volume_claim.home.metadata.0.name + read_only = false + } + } - affinity { - pod_anti_affinity { - // This affinity attempts to spread out all workspace pods evenly across - // nodes. - preferred_during_scheduling_ignored_during_execution { - weight = 1 - pod_affinity_term { - topology_key = "kubernetes.io/hostname" - label_selector { - match_expressions { - key = "app.kubernetes.io/name" - operator = "In" - values = ["coder-workspace"] + affinity { + // This affinity attempts to spread out all workspace pods evenly across + // nodes. + pod_anti_affinity { + preferred_during_scheduling_ignored_during_execution { + weight = 1 + pod_affinity_term { + topology_key = "kubernetes.io/hostname" + label_selector { + match_expressions { + key = "app.kubernetes.io/name" + operator = "In" + values = ["coder-workspace"] + } + } } } }