-
Notifications
You must be signed in to change notification settings - Fork 887
feat(helm/provisioner): support deploying multiple provisioners in same namespace #15637
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -32,5 +32,118 @@ coder: | |
value: "0.0.0.0:2112" | ||
replicaCount: 10 | ||
provisionerDaemon: | ||
pskSecretName: "coder-provisioner-psk" | ||
keySecretName: "coder-provisionerd-key" | ||
keySecretKey: "provisionerd-key" | ||
Comment on lines
-35
to
+36
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. self-review: advertising provisioner keys as the preferred auth method |
||
``` | ||
|
||
## Specific Examples | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. self-review: these examples are essentially lifted from our tests |
||
|
||
Below are some common specific use-cases when deploying a Coder provisioner. | ||
|
||
### Set Labels and Annotations | ||
|
||
If you need to set deployment- or pod-level labels and annotations, set `coder.{annotations,labels}` or `coder.{podAnnotations,podLabels}`. | ||
|
||
Example: | ||
|
||
```yaml | ||
coder: | ||
# ... | ||
annotations: | ||
com.coder/annotation/foo: bar | ||
com.coder/annotation/baz: qux | ||
labels: | ||
com.coder/label/foo: bar | ||
com.coder/label/baz: qux | ||
podAnnotations: | ||
com.coder/podAnnotation/foo: bar | ||
com.coder/podAnnotation/baz: qux | ||
podLabels: | ||
com.coder/podLabel/foo: bar | ||
com.coder/podLabel/baz: qux | ||
``` | ||
|
||
### Additional Templates | ||
|
||
You can include extra Kubernetes manifests in `extraTemplates`. | ||
|
||
The below example will also create a `ConfigMap` along with the Helm release: | ||
|
||
```yaml | ||
coder: | ||
# ... | ||
provisionerDaemon: | ||
# ... | ||
extraTemplates: | ||
- | | ||
apiVersion: v1 | ||
kind: ConfigMap | ||
metadata: | ||
name: some-config | ||
namespace: {{ .Release.Namespace }} | ||
data: | ||
key: some-value | ||
``` | ||
|
||
### Disable Service Account Creation | ||
|
||
### Deploying multiple provisioners in the same namespace | ||
|
||
To deploy multiple provisioners in the same namespace, set the following values explicitly to avoid conflicts: | ||
|
||
- `nameOverride`: controls the name of the provisioner deployment | ||
- `serviceAccount.name`: controls the name of the service account. | ||
|
||
Note that `nameOverride` does not apply to `extraTemplates`, as illustrated below: | ||
|
||
```yaml | ||
coder: | ||
# ... | ||
serviceAccount: | ||
name: other-coder-provisioner | ||
provisionerDaemon: | ||
# ... | ||
nameOverride: "other-coder-provisioner" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. self-review: this setting is essentially buried in |
||
extraTemplates: | ||
- | | ||
apiVersion: v1 | ||
kind: ConfigMap | ||
metadata: | ||
name: some-other-config | ||
namespace: {{ .Release.Namespace }} | ||
data: | ||
key: some-other-value | ||
Comment on lines
+107
to
+115
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. self-review: this isn't strictly necessary, but I figured it could be a potential point of confusion, so elected to clarify it here. |
||
``` | ||
|
||
If you wish to deploy a second provisioner that references an existing service account, you can do so as follows: | ||
|
||
- Set `coder.serviceAccount.disableCreate=true` to disable service account creation, | ||
- Set `coder.serviceAccount.workspacePerms=false` to disable creation of a role and role binding, | ||
- Set `coder.serviceAccount.nameOverride` to the name of an existing service account. | ||
|
||
See below for a concrete example: | ||
|
||
```yaml | ||
coder: | ||
# ... | ||
serviceAccount: | ||
name: preexisting-service-account | ||
disableCreate: true | ||
workspacePerms: false | ||
provisionerDaemon: | ||
# ... | ||
nameOverride: "other-coder-provisioner" | ||
``` | ||
|
||
# Testing | ||
|
||
The test suite for this chart lives in `./tests/chart_test.go`. | ||
|
||
Each test case runs `helm template` against the corresponding `test_case.yaml`, and compares the output with that of the corresponding `test_case.golden` in `./tests/testdata`. | ||
If `expectedError` is not empty for that specific test case, no corresponding `.golden` file is required. | ||
|
||
To add a new test case: | ||
|
||
- Create an appropriately named `.yaml` file in `testdata/` along with a corresponding `.golden` file, if required. | ||
- Add the test case to the array in `chart_test.go`, setting `name` to the name of the files you added previously (without the extension). If appropriate, set `expectedError`. | ||
- Run the tests and ensure that no regressions in existing test cases occur: `go test ./tests`. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,7 @@ | ||
--- | ||
{{- if not .Values.coder.serviceAccount.disableCreate }} | ||
{{ include "libcoder.serviceaccount" (list . "coder.serviceaccount") }} | ||
{{- end }} | ||
Comment on lines
+2
to
+4
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. self-review: this was added for I'm not sure if it would be better to do it in libcoder, folks don't seem to look in there much. |
||
|
||
--- | ||
{{ include "libcoder.deployment" (list . "coder.deployment") }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,144 @@ | ||
--- | ||
# Source: coder-provisioner/templates/coder.yaml | ||
apiVersion: v1 | ||
kind: ServiceAccount | ||
metadata: | ||
annotations: {} | ||
labels: | ||
app.kubernetes.io/instance: release-name | ||
app.kubernetes.io/managed-by: Helm | ||
app.kubernetes.io/name: other-coder-provisioner | ||
app.kubernetes.io/part-of: other-coder-provisioner | ||
app.kubernetes.io/version: 0.1.0 | ||
helm.sh/chart: coder-provisioner-0.1.0 | ||
name: other-coder-provisioner | ||
--- | ||
# Source: coder-provisioner/templates/extra-templates.yaml | ||
apiVersion: v1 | ||
kind: ConfigMap | ||
metadata: | ||
name: some-config | ||
namespace: default | ||
data: | ||
key: some-value | ||
--- | ||
# Source: coder-provisioner/templates/rbac.yaml | ||
apiVersion: rbac.authorization.k8s.io/v1 | ||
kind: Role | ||
metadata: | ||
name: other-coder-provisioner-workspace-perms | ||
rules: | ||
- apiGroups: [""] | ||
resources: ["pods"] | ||
verbs: | ||
- create | ||
- delete | ||
- deletecollection | ||
- get | ||
- list | ||
- patch | ||
- update | ||
- watch | ||
- apiGroups: [""] | ||
resources: ["persistentvolumeclaims"] | ||
verbs: | ||
- create | ||
- delete | ||
- deletecollection | ||
- get | ||
- list | ||
- patch | ||
- update | ||
- watch | ||
- apiGroups: | ||
- apps | ||
resources: | ||
- deployments | ||
verbs: | ||
- create | ||
- delete | ||
- deletecollection | ||
- get | ||
- list | ||
- patch | ||
- update | ||
- watch | ||
--- | ||
# Source: coder-provisioner/templates/rbac.yaml | ||
apiVersion: rbac.authorization.k8s.io/v1 | ||
kind: RoleBinding | ||
metadata: | ||
name: "other-coder-provisioner" | ||
subjects: | ||
- kind: ServiceAccount | ||
name: "other-coder-provisioner" | ||
roleRef: | ||
apiGroup: rbac.authorization.k8s.io | ||
kind: Role | ||
name: other-coder-provisioner-workspace-perms | ||
--- | ||
# Source: coder-provisioner/templates/coder.yaml | ||
apiVersion: apps/v1 | ||
kind: Deployment | ||
metadata: | ||
annotations: {} | ||
labels: | ||
app.kubernetes.io/instance: release-name | ||
app.kubernetes.io/managed-by: Helm | ||
app.kubernetes.io/name: other-coder-provisioner | ||
app.kubernetes.io/part-of: other-coder-provisioner | ||
app.kubernetes.io/version: 0.1.0 | ||
helm.sh/chart: coder-provisioner-0.1.0 | ||
name: other-coder-provisioner | ||
spec: | ||
replicas: 1 | ||
selector: | ||
matchLabels: | ||
app.kubernetes.io/instance: release-name | ||
app.kubernetes.io/name: other-coder-provisioner | ||
template: | ||
metadata: | ||
annotations: {} | ||
labels: | ||
app.kubernetes.io/instance: release-name | ||
app.kubernetes.io/managed-by: Helm | ||
app.kubernetes.io/name: other-coder-provisioner | ||
app.kubernetes.io/part-of: other-coder-provisioner | ||
app.kubernetes.io/version: 0.1.0 | ||
helm.sh/chart: coder-provisioner-0.1.0 | ||
spec: | ||
containers: | ||
- args: | ||
- provisionerd | ||
- start | ||
command: | ||
- /opt/coder | ||
env: | ||
- name: CODER_PROMETHEUS_ADDRESS | ||
value: 0.0.0.0:2112 | ||
- name: CODER_PROVISIONER_DAEMON_PSK | ||
valueFrom: | ||
secretKeyRef: | ||
key: psk | ||
name: coder-provisioner-psk | ||
- name: CODER_URL | ||
value: http://coder.default.svc.cluster.local | ||
image: ghcr.io/coder/coder:latest | ||
imagePullPolicy: IfNotPresent | ||
lifecycle: {} | ||
name: coder | ||
ports: null | ||
resources: {} | ||
securityContext: | ||
allowPrivilegeEscalation: false | ||
readOnlyRootFilesystem: null | ||
runAsGroup: 1000 | ||
runAsNonRoot: true | ||
runAsUser: 1000 | ||
seccompProfile: | ||
type: RuntimeDefault | ||
volumeMounts: [] | ||
restartPolicy: Always | ||
serviceAccountName: other-coder-provisioner | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. self-review: the fact that |
||
terminationGracePeriodSeconds: 600 | ||
volumes: [] |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
coder: | ||
image: | ||
tag: latest | ||
serviceAccount: | ||
name: other-coder-provisioner | ||
nameOverride: "other-coder-provisioner" | ||
# Note that extraTemplates does not respect nameOverride. | ||
extraTemplates: | ||
- | | ||
apiVersion: v1 | ||
kind: ConfigMap | ||
metadata: | ||
name: some-config | ||
namespace: {{ .Release.Namespace }} | ||
data: | ||
key: some-value |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,67 @@ | ||
--- | ||
# Source: coder-provisioner/templates/coder.yaml | ||
--- | ||
apiVersion: apps/v1 | ||
kind: Deployment | ||
metadata: | ||
annotations: {} | ||
labels: | ||
app.kubernetes.io/instance: release-name | ||
app.kubernetes.io/managed-by: Helm | ||
app.kubernetes.io/name: other-coder-provisioner | ||
app.kubernetes.io/part-of: other-coder-provisioner | ||
app.kubernetes.io/version: 0.1.0 | ||
helm.sh/chart: coder-provisioner-0.1.0 | ||
name: other-coder-provisioner | ||
spec: | ||
replicas: 1 | ||
selector: | ||
matchLabels: | ||
app.kubernetes.io/instance: release-name | ||
app.kubernetes.io/name: other-coder-provisioner | ||
template: | ||
metadata: | ||
annotations: {} | ||
labels: | ||
app.kubernetes.io/instance: release-name | ||
app.kubernetes.io/managed-by: Helm | ||
app.kubernetes.io/name: other-coder-provisioner | ||
app.kubernetes.io/part-of: other-coder-provisioner | ||
app.kubernetes.io/version: 0.1.0 | ||
helm.sh/chart: coder-provisioner-0.1.0 | ||
spec: | ||
containers: | ||
- args: | ||
- provisionerd | ||
- start | ||
command: | ||
- /opt/coder | ||
env: | ||
- name: CODER_PROMETHEUS_ADDRESS | ||
value: 0.0.0.0:2112 | ||
- name: CODER_PROVISIONER_DAEMON_PSK | ||
valueFrom: | ||
secretKeyRef: | ||
key: psk | ||
name: coder-provisioner-psk | ||
- name: CODER_URL | ||
value: http://coder.default.svc.cluster.local | ||
image: ghcr.io/coder/coder:latest | ||
imagePullPolicy: IfNotPresent | ||
lifecycle: {} | ||
name: coder | ||
ports: null | ||
resources: {} | ||
securityContext: | ||
allowPrivilegeEscalation: false | ||
readOnlyRootFilesystem: null | ||
runAsGroup: 1000 | ||
runAsNonRoot: true | ||
runAsUser: 1000 | ||
seccompProfile: | ||
type: RuntimeDefault | ||
volumeMounts: [] | ||
restartPolicy: Always | ||
serviceAccountName: existing-coder-provisioner-serviceaccount | ||
terminationGracePeriodSeconds: 600 | ||
volumes: [] |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
coder: | ||
image: | ||
tag: latest | ||
serviceAccount: | ||
name: "existing-coder-provisioner-serviceaccount" | ||
disableCreate: true | ||
workspacePerms: false | ||
nameOverride: "other-coder-provisioner" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self review: drive-by typo fix