Skip to content

chore(examples): update kubernetes devcontainer template with envbuilder provider #14267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 15, 2024
Merged

chore(examples): update kubernetes devcontainer template with envbuilder provider #14267

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d5e2fa0
chore(examples): update kubernetes devcontainer template with envbuil…
johnstcn Aug 14, 2024
821ea40
make insecure a template variable
johnstcn Aug 14, 2024
9d94cfc
Update examples/templates/devcontainer-kubernetes/README.md
johnstcn Aug 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions examples/templates/devcontainer-kubernetes/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ Provision Devcontainers as [Coder workspaces](https://coder.com/docs/workspaces)

**Container Image**: This template uses the [envbuilder image](https://github.com/coder/envbuilder) to build a Devcontainer from a `devcontainer.json`.

**(Optional) Cache Registry**: Envbuilder can utilize a Docker registry as a cache to speed up workspace builds. The [envbuilder Terraform provider](https://github.com/coder/terraform-provider-envbuilder) will check the contents of the cache to determine if a prebuilt image exists. In the case of some missing layers in the registry (partial cache miss), Envbuilder can still utilize some of the build cache from the registry.

### Authentication

This template authenticates using a `~/.kube/config`, if present on the server, or via built-in authentication if the Coder provisioner is running on Kubernetes with an authorized ServiceAccount. To use another [authentication method](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#authentication), edit the template.
Expand All @@ -31,6 +33,7 @@ This template provisions the following resources:

- Kubernetes deployment (ephemeral)
- Kubernetes persistent volume claim (persistent on `/workspaces`)
- Envbuilder cached image (optional, persistent).

This template will fetch a Git repo containing a `devcontainer.json` specified by the `repo` parameter, and builds it
with [`envbuilder`](https://github.com/coder/envbuilder).
Expand All @@ -47,6 +50,8 @@ Edit the `devcontainer.json` instead!
To speed up your builds, you can use a container registry as a cache.
When creating the template, set the parameter `cache_repo`.

See the [Envbuilder Terraform Provider Examples](https://github.com/coder/terraform-provider-envbuilder/blob/main/examples/resources/envbuilder_cached_image/envbuilder_cached_image_resource.tf/) for a more complete example of how the provider works.

> [!NOTE] We recommend using a registry cache with authentication enabled.
> To allow Envbuilder to authenticate with the registry cache, specify the variable `cache_repo_dockerconfig_secret`
> with the name of a Kubernetes secret in the same namespace as Coder. The secret must contain the key `.dockerconfigjson`.
90 changes: 75 additions & 15 deletions examples/templates/devcontainer-kubernetes/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ terraform {
kubernetes = {
source = "hashicorp/kubernetes"
}
envbuilder = {
source = "coder/envbuilder"
}
}
}

Expand All @@ -15,6 +18,7 @@ provider "kubernetes" {
# Authenticate via ~/.kube/config or a Coder-specific ServiceAccount, depending on admin preferences
config_path = var.use_kubeconfig == true ? "~/.kube/config" : null
}
provider "envbuilder" {}

data "coder_provisioner" "me" {}
data "coder_workspace" "me" {}
Expand Down Expand Up @@ -43,10 +47,15 @@ variable "namespace" {
variable "cache_repo" {
default = ""
description = "Use a container registry as a cache to speed up builds."
sensitive = true
type = string
}

variable "insecure_cache_repo" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice solution 👍🏻

default = false
description = "Enable this option if your cache registry does not serve HTTPS."
type = bool
}

data "coder_parameter" "cpu" {
type = "number"
name = "cpu"
Expand Down Expand Up @@ -139,20 +148,45 @@ data "kubernetes_secret" "cache_repo_dockerconfig_secret" {
}

locals {
deployment_name = "coder-${data.coder_workspace_owner.me.name}-${lower(data.coder_workspace.me.name)}"
deployment_name = "coder-${lower(data.coder_workspace.me.id)}"
devcontainer_builder_image = data.coder_parameter.devcontainer_builder.value
git_author_name = coalesce(data.coder_workspace_owner.me.full_name, data.coder_workspace_owner.me.name)
git_author_email = data.coder_workspace_owner.me.email
repo_url = data.coder_parameter.repo.value
# The envbuilder provider requires a key-value map of environment variables.
envbuilder_env = {
"CODER_AGENT_TOKEN" : coder_agent.main.token,
# Use the docker gateway if the access URL is 127.0.0.1
"CODER_AGENT_URL" : replace(data.coder_workspace.me.access_url, "/localhost|127\\.0\\.0\\.1/", "host.docker.internal"),
"ENVBUILDER_GIT_URL" : local.repo_url,
# Use the docker gateway if the access URL is 127.0.0.1
"ENVBUILDER_INIT_SCRIPT" : replace(coder_agent.main.init_script, "/localhost|127\\.0\\.0\\.1/", "host.docker.internal"),
"ENVBUILDER_FALLBACK_IMAGE" : data.coder_parameter.fallback_image.value,
"ENVBUILDER_CACHE_REPO" : var.cache_repo,
"ENVBUILDER_DOCKER_CONFIG_BASE64" : try(data.kubernetes_secret.cache_repo_dockerconfig_secret[0].data[".dockerconfigjson"], ""),
"ENVBUILDER_PUSH_IMAGE" : var.cache_repo == "" ? "" : "true",
"ENVBUILDER_INSECURE" : "${var.insecure_cache_repo}",
}
}

# Check for the presence of a prebuilt image in the cache repo
# that we can use instead.
resource "envbuilder_cached_image" "cached" {
count = var.cache_repo == "" ? 0 : data.coder_workspace.me.start_count
builder_image = local.devcontainer_builder_image
git_url = local.repo_url
cache_repo = var.cache_repo
extra_env = local.envbuilder_env
insecure = var.insecure_cache_repo
}

resource "kubernetes_persistent_volume_claim" "home" {
resource "kubernetes_persistent_volume_claim" "workspaces" {
metadata {
name = "coder-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}-home"
name = "coder-${lower(data.coder_workspace.me.id)}-workspaces"
namespace = var.namespace
labels = {
"app.kubernetes.io/name" = "coder-pvc"
"app.kubernetes.io/instance" = "coder-pvc-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}"
"app.kubernetes.io/name" = "coder-${lower(data.coder_workspace.me.id)}-workspaces"
"app.kubernetes.io/instance" = "coder-${lower(data.coder_workspace.me.id)}-workspaces"
"app.kubernetes.io/part-of" = "coder"
//Coder-specific labels.
"com.coder.resource" = "true"
Expand All @@ -173,13 +207,14 @@ resource "kubernetes_persistent_volume_claim" "home" {
storage = "${data.coder_parameter.workspaces_volume_size.value}Gi"
}
}
# storage_class_name = "local-path" # Configure the StorageClass to use here, if required.
}
}

resource "kubernetes_deployment" "main" {
count = data.coder_workspace.me.start_count
depends_on = [
kubernetes_persistent_volume_claim.home
kubernetes_persistent_volume_claim.workspaces
]
wait_for_rollout = false
metadata {
Expand Down Expand Up @@ -222,7 +257,7 @@ resource "kubernetes_deployment" "main" {

container {
name = "dev"
image = local.devcontainer_builder_image
image = var.cache_repo == "" ? local.devcontainer_builder_image : envbuilder_cached_image.cached.0.image
image_pull_policy = "Always"
security_context {}
env {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review: leaving as-is until coder/terraform-provider-envbuilder#31 is resolved.

There may be some additional Terraform jiggery pokery requried to convert local.envbuilder_env into a block list.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A for-loop would be nice here to set all the envs, wdyt?

Right now it can come from locals.envbuilder_env, but eventually from envbuilder_cached_image.cached.0.env.

env {
	for_each = locals.envbuilder_env
	name = each.key
	value = each.value
}

Not sure if tomap is required (for_each = tomap(locals.envbuilder_env)), probably not?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but then that needs to be referenced in the kubernetes_pod spec below. I think that may need a dynamic{} block?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, my tf foo is not strong enough 😄

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll address this in a follow-up. The current way the envbuilder provider outputs the computed env isn't really conducive to this scenario right now anyway.

Expand All @@ -249,6 +284,14 @@ resource "kubernetes_deployment" "main" {
name = "ENVBUILDER_CACHE_REPO"
value = var.cache_repo
}
env {
name = "ENVBUILDER_PUSH_IMAGE"
value = var.cache_repo == "" ? "" : "true"
}
env {
name = "ENVBUILDER_INSECURE"
value = var.insecure_cache_repo
}
env {
name = "ENVBUILDER_DOCKER_CONFIG_BASE64"
value = try(data.kubernetes_secret.cache_repo_dockerconfig_secret[0].data[".dockerconfigjson"], "")
Expand All @@ -271,16 +314,16 @@ resource "kubernetes_deployment" "main" {
}
}
volume_mount {
mount_path = "/home/coder"
name = "home"
mount_path = "/workspaces"
name = "workspaces"
read_only = false
}
}

volume {
name = "home"
name = "workspaces"
persistent_volume_claim {
claim_name = kubernetes_persistent_volume_claim.home.metadata.0.name
claim_name = kubernetes_persistent_volume_claim.workspaces.metadata.0.name
read_only = false
}
}
Expand Down Expand Up @@ -357,9 +400,9 @@ resource "coder_agent" "main" {
}

metadata {
display_name = "Home Disk"
key = "3_home_disk"
script = "coder stat disk --path $HOME"
display_name = "Workspaces Disk"
key = "3_workspaces_disk"
script = "coder stat disk --path /workspaces"
interval = 60
timeout = 1
}
Expand Down Expand Up @@ -417,3 +460,20 @@ resource "coder_app" "code-server" {
threshold = 6
}
}

resource "coder_metadata" "container_info" {
count = data.coder_workspace.me.start_count
resource_id = coder_agent.main.id
item {
key = "workspace image"
value = var.cache_repo == "" ? local.devcontainer_builder_image : envbuilder_cached_image.cached.0.image
}
item {
key = "git url"
value = local.repo_url
}
item {
key = "cache repo"
value = var.cache_repo == "" ? "not enabled" : var.cache_repo
}
}
Loading