Skip to content

feat(helm/provisioner): support deploying multiple provisioners in same namespace #15637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion helm/coder/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ coder:
annotations: {}
# coder.serviceAccount.name -- The service account name
name: coder
# coder.serviceAccount.name -- Whether to create the service account or use existing service account
# coder.serviceAccount.disableCreate -- Whether to create the service account or use existing service account.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self review: drive-by typo fix

disableCreate: false

# coder.securityContext -- Fields related to the container's security
Expand Down
115 changes: 114 additions & 1 deletion helm/provisioner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,118 @@ coder:
value: "0.0.0.0:2112"
replicaCount: 10
provisionerDaemon:
pskSecretName: "coder-provisioner-psk"
keySecretName: "coder-provisionerd-key"
keySecretKey: "provisionerd-key"
Comment on lines -35 to +36
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self-review: advertising provisioner keys as the preferred auth method

```

## Specific Examples
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self-review: these examples are essentially lifted from our tests


Below are some common specific use-cases when deploying a Coder provisioner.

### Set Labels and Annotations

If you need to set deployment- or pod-level labels and annotations, set `coder.{annotations,labels}` or `coder.{podAnnotations,podLabels}`.

Example:

```yaml
coder:
# ...
annotations:
com.coder/annotation/foo: bar
com.coder/annotation/baz: qux
labels:
com.coder/label/foo: bar
com.coder/label/baz: qux
podAnnotations:
com.coder/podAnnotation/foo: bar
com.coder/podAnnotation/baz: qux
podLabels:
com.coder/podLabel/foo: bar
com.coder/podLabel/baz: qux
```

### Additional Templates

You can include extra Kubernetes manifests in `extraTemplates`.

The below example will also create a `ConfigMap` along with the Helm release:

```yaml
coder:
# ...
provisionerDaemon:
# ...
extraTemplates:
- |
apiVersion: v1
kind: ConfigMap
metadata:
name: some-config
namespace: {{ .Release.Namespace }}
data:
key: some-value
```

### Disable Service Account Creation

### Deploying multiple provisioners in the same namespace

To deploy multiple provisioners in the same namespace, set the following values explicitly to avoid conflicts:

- `nameOverride`: controls the name of the provisioner deployment
- `serviceAccount.name`: controls the name of the service account.

Note that `nameOverride` does not apply to `extraTemplates`, as illustrated below:

```yaml
coder:
# ...
serviceAccount:
name: other-coder-provisioner
provisionerDaemon:
# ...
nameOverride: "other-coder-provisioner"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self-review: this setting is essentially buried in libcoder, but it seems to be the best way to do this. Introducing a new separate variable here is just going to make things more complicated.

extraTemplates:
- |
apiVersion: v1
kind: ConfigMap
metadata:
name: some-other-config
namespace: {{ .Release.Namespace }}
data:
key: some-other-value
Comment on lines +107 to +115
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self-review: this isn't strictly necessary, but I figured it could be a potential point of confusion, so elected to clarify it here.

```

If you wish to deploy a second provisioner that references an existing service account, you can do so as follows:

- Set `coder.serviceAccount.disableCreate=true` to disable service account creation,
- Set `coder.serviceAccount.workspacePerms=false` to disable creation of a role and role binding,
- Set `coder.serviceAccount.nameOverride` to the name of an existing service account.

See below for a concrete example:

```yaml
coder:
# ...
serviceAccount:
name: preexisting-service-account
disableCreate: true
workspacePerms: false
provisionerDaemon:
# ...
nameOverride: "other-coder-provisioner"
```

# Testing

The test suite for this chart lives in `./tests/chart_test.go`.

Each test case runs `helm template` against the corresponding `test_case.yaml`, and compares the output with that of the corresponding `test_case.golden` in `./tests/testdata`.
If `expectedError` is not empty for that specific test case, no corresponding `.golden` file is required.

To add a new test case:

- Create an appropriately named `.yaml` file in `testdata/` along with a corresponding `.golden` file, if required.
- Add the test case to the array in `chart_test.go`, setting `name` to the name of the files you added previously (without the extension). If appropriate, set `expectedError`.
- Run the tests and ensure that no regressions in existing test cases occur: `go test ./tests`.
2 changes: 2 additions & 0 deletions helm/provisioner/templates/coder.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
---
{{- if not .Values.coder.serviceAccount.disableCreate }}
{{ include "libcoder.serviceaccount" (list . "coder.serviceaccount") }}
{{- end }}
Comment on lines +2 to +4
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self-review: this was added for helm/coder in #14817 but not ported over here.

I'm not sure if it would be better to do it in libcoder, folks don't seem to look in there much.


---
{{ include "libcoder.deployment" (list . "coder.deployment") }}
12 changes: 12 additions & 0 deletions helm/provisioner/tests/chart_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,18 @@ var testCases = []testCase{
name: "extra_templates",
expectedError: "",
},
{
name: "sa_disabled",
expectedError: "",
},
{
name: "name_override",
expectedError: "",
},
{
name: "name_override_existing_sa",
expectedError: "",
},
}

type testCase struct {
Expand Down
144 changes: 144 additions & 0 deletions helm/provisioner/tests/testdata/name_override.golden
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
---
# Source: coder-provisioner/templates/coder.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
annotations: {}
labels:
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: other-coder-provisioner
app.kubernetes.io/part-of: other-coder-provisioner
app.kubernetes.io/version: 0.1.0
helm.sh/chart: coder-provisioner-0.1.0
name: other-coder-provisioner
---
# Source: coder-provisioner/templates/extra-templates.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: some-config
namespace: default
data:
key: some-value
---
# Source: coder-provisioner/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: other-coder-provisioner-workspace-perms
rules:
- apiGroups: [""]
resources: ["pods"]
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- apps
resources:
- deployments
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
---
# Source: coder-provisioner/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: "other-coder-provisioner"
subjects:
- kind: ServiceAccount
name: "other-coder-provisioner"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: other-coder-provisioner-workspace-perms
---
# Source: coder-provisioner/templates/coder.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations: {}
labels:
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: other-coder-provisioner
app.kubernetes.io/part-of: other-coder-provisioner
app.kubernetes.io/version: 0.1.0
helm.sh/chart: coder-provisioner-0.1.0
name: other-coder-provisioner
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: release-name
app.kubernetes.io/name: other-coder-provisioner
template:
metadata:
annotations: {}
labels:
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: other-coder-provisioner
app.kubernetes.io/part-of: other-coder-provisioner
app.kubernetes.io/version: 0.1.0
helm.sh/chart: coder-provisioner-0.1.0
spec:
containers:
- args:
- provisionerd
- start
command:
- /opt/coder
env:
- name: CODER_PROMETHEUS_ADDRESS
value: 0.0.0.0:2112
- name: CODER_PROVISIONER_DAEMON_PSK
valueFrom:
secretKeyRef:
key: psk
name: coder-provisioner-psk
- name: CODER_URL
value: http://coder.default.svc.cluster.local
image: ghcr.io/coder/coder:latest
imagePullPolicy: IfNotPresent
lifecycle: {}
name: coder
ports: null
resources: {}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: null
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
volumeMounts: []
restartPolicy: Always
serviceAccountName: other-coder-provisioner
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self-review: the fact that nameOverride has no bearing on serviceAccountName explicitly allows us to reference a pre-existing service account

terminationGracePeriodSeconds: 600
volumes: []
16 changes: 16 additions & 0 deletions helm/provisioner/tests/testdata/name_override.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
coder:
image:
tag: latest
serviceAccount:
name: other-coder-provisioner
nameOverride: "other-coder-provisioner"
# Note that extraTemplates does not respect nameOverride.
extraTemplates:
- |
apiVersion: v1
kind: ConfigMap
metadata:
name: some-config
namespace: {{ .Release.Namespace }}
data:
key: some-value
67 changes: 67 additions & 0 deletions helm/provisioner/tests/testdata/name_override_existing_sa.golden
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
# Source: coder-provisioner/templates/coder.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations: {}
labels:
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: other-coder-provisioner
app.kubernetes.io/part-of: other-coder-provisioner
app.kubernetes.io/version: 0.1.0
helm.sh/chart: coder-provisioner-0.1.0
name: other-coder-provisioner
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: release-name
app.kubernetes.io/name: other-coder-provisioner
template:
metadata:
annotations: {}
labels:
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: other-coder-provisioner
app.kubernetes.io/part-of: other-coder-provisioner
app.kubernetes.io/version: 0.1.0
helm.sh/chart: coder-provisioner-0.1.0
spec:
containers:
- args:
- provisionerd
- start
command:
- /opt/coder
env:
- name: CODER_PROMETHEUS_ADDRESS
value: 0.0.0.0:2112
- name: CODER_PROVISIONER_DAEMON_PSK
valueFrom:
secretKeyRef:
key: psk
name: coder-provisioner-psk
- name: CODER_URL
value: http://coder.default.svc.cluster.local
image: ghcr.io/coder/coder:latest
imagePullPolicy: IfNotPresent
lifecycle: {}
name: coder
ports: null
resources: {}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: null
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
volumeMounts: []
restartPolicy: Always
serviceAccountName: existing-coder-provisioner-serviceaccount
terminationGracePeriodSeconds: 600
volumes: []
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
coder:
image:
tag: latest
serviceAccount:
name: "existing-coder-provisioner-serviceaccount"
disableCreate: true
workspacePerms: false
nameOverride: "other-coder-provisioner"
Loading
Loading