Skip to content

K8s template uses an authenticated environment #2104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 6, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 52 additions & 56 deletions examples/templates/kubernetes-multi-service/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,75 +4,71 @@ description: Get started with Kubernetes development.
tags: [cloud, kubernetes]
---

# Authentication
# Getting started

This template features two ways to authenticate to a Kubernetes cluster.
## RBAC

## kubeconfig (Coder host)
The Coder provisioner requires permission to administer pods to use this template. The template
creates workspaces in a single Kubernetes namespace, using the `workspaces_namespace` parameter set
while creating the template.

If the Coder host has a local `~/.kube/config`, you can use this to authenticate
with Coder. Make sure this is done with same user that's running the `coder` service.
Create a role as follows and bind it to the user or service account that runs the coder host.

## ServiceAccount
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: coder
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["*"]
```

Create a ServiceAccount and role on your cluster to authenticate your template with Coder.
## Authentication

1. Run the following command on a device with Kubernetes context:
This template can authenticate using in-cluster authentication, or using a kubeconfig local to the
Coder host. For additional authentication options, consult the [Kubernetes provider
documentation](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs).

```sh
CODER_NAMESPACE=default
kubectl apply -n $CODER_NAMESPACE -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: coder
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: coder
rules:
- apiGroups: ["", "apps", "networking.k8s.io"] # "" indicates the core API group
resources: ["persistentvolumeclaims", "pods", "deployments", "services", "secrets", "pods/exec","pods/log", "events", "networkpolicies", "serviceaccounts"]
verbs: ["create", "get", "list", "watch", "update", "patch", "delete", "deletecollection"]
- apiGroups: ["metrics.k8s.io", "storage.k8s.io"]
resources: ["pods", "storageclasses"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: coder
subjects:
- kind: ServiceAccount
name: coder
roleRef:
kind: Role
name: coder
apiGroup: rbac.authorization.k8s.io
EOF
```
### kubeconfig on Coder host

1. Use the following commands to fetch the values:
If the Coder host has a local `~/.kube/config`, you can use this to authenticate
with Coder. Make sure this is done with same user that's running the `coder` service.

To use this authentication, set the parameter `use_kubeconfig` to true.

**Cluster IP:**
### In-cluster authentication

```sh
kubectl cluster-info | grep "control plane"
```
If the Coder host runs in a Pod on the same Kubernetes cluster as you are creating workspaces in,
you can use in-cluster authentication.

**CA certificate**
To use this authentication, set the parameter `use_kubeconfig` to false.

```sh
kubectl get secrets -n $CODER_NAMESPACE -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='coder')].data['ca\.crt']}{'\n'}"
```
The Terraform provisioner will automatically use the service account associated with the pod to
authenticate to Kubernetes. Be sure to bind a [role with appropriate permission](#rbac) to the
service account. For example, assuming the Coder host runs in the same namespace as you intend
to create workspaces:

**Token**
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: coder

```sh
kubectl get secrets -n $CODER_NAMESPACE -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='coder')].data['token']}{'\n'}"
```
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: coder
subjects:
- kind: ServiceAccount
name: coder
roleRef:
kind: Role
name: coder
apiGroup: rbac.authorization.k8s.io
```

**Namespace**
Then start the Coder host with `serviceAccountName: coder` in the pod spec.

This should be the same as `$CODER_NAMESPACE`, set in step 1.
61 changes: 12 additions & 49 deletions examples/templates/kubernetes-multi-service/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -11,70 +11,32 @@ terraform {
}
}

variable "step1_use_kubeconfig" {
variable "use_kubeconfig" {
type = bool
sensitive = true
description = <<-EOF
Use host kubeconfig? (true/false)

If true, a valid "~/.kube/config" must be present on the Coder host. This
is likely not your local machine unless you are using `coder server --dev.`

If false, proceed for instructions creating a ServiceAccount on your existing
Kubernetes cluster.
EOF
}

variable "step2_cluster_host" {
type = string
sensitive = true
description = <<-EOF
Hint: You can use:
$ kubectl cluster-info | grep "control plane"

Set this to false if the Coder host is itself running as a Pod on the same
Kubernetes cluster as you are deploying workspaces to.

Leave blank if using ~/.kube/config (from step 1)
EOF
}

variable "step3_certificate" {
type = string
sensitive = true
description = <<-EOF
Use docs at https://github.com/coder/coder/tree/main/examples/templates/kubernetes-multi-service#serviceaccount to create a ServiceAccount for Coder and grab values.

Enter CA certificate

Leave blank if using ~/.kube/config (from step 1)
EOF
}

variable "step4_token" {
type = string
sensitive = true
description = <<-EOF
Enter token (refer to docs at https://github.com/coder/coder/tree/main/examples/templates/kubernetes-multi-service#serviceaccount)
Set this to true if the Coder host is running outside the Kubernetes cluster
for workspaces. A valid "~/.kube/config" must be present on the Coder host. This
is likely not your local machine unless you are using `coder server --dev.`

Leave blank if using ~/.kube/config (from step 1)
EOF
}

variable "step5_coder_namespace" {
variable "workspaces_namespace" {
type = string
sensitive = true
description = <<-EOF
Enter namespace (refer to docs at https://github.com/coder/coder/tree/main/examples/templates/kubernetes-multi-service#serviceaccount)

Leave blank if using ~/.kube/config (from step 1)
EOF
description = "The namespace to create workspaces in (must exist prior to creating workspaces)"
default = "coder-workspaces"
}

provider "kubernetes" {
# Authenticate via ~/.kube/config or a Coder-specific ServiceAccount, depending on admin preferences
config_path = var.step1_use_kubeconfig == true ? "~/.kube/config" : null
host = var.step1_use_kubeconfig == false ? var.step2_cluster_host : null
cluster_ca_certificate = var.step1_use_kubeconfig == false ? base64decode(var.step3_certificate) : null
token = var.step1_use_kubeconfig == false ? base64decode(var.step4_token) : null
config_path = var.use_kubeconfig == true ? "~/.kube/config" : null
}

data "coder_workspace" "me" {}
Expand All @@ -97,7 +59,8 @@ resource "coder_agent" "ubuntu" {
resource "kubernetes_pod" "main" {
count = data.coder_workspace.me.start_count
metadata {
name = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
name = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
namespace = var.workspaces_namespace
}
spec {
container {
Expand Down