Increase Timeout from 5 Minutes (K8s pods orphaned) #15909

groundsada · 2024-12-18T00:12:24Z

groundsada
Dec 18, 2024

Hi,

We are running Coder on our K8s cluster. Some templates use big images, and it takes more than 5 minutes to pull them. Coder times out after 5 minutes, showing that the workspace has failed. The pod keeps starting on K8s and doesn't show as healthy or started on Coder. This leads to orphaned pods that are in fact running (and wasting resources) without users knowing that their pods started.

Ask:

Is there a way to escape this behavior with Terraform?
Is there a way to increase the timeout from 5 minutes to a bigger threshold?

Answered by matifali

Dec 18, 2024

Could you try using timeouts then?

View full answer

matifali · 2024-12-18T06:31:34Z

matifali
Dec 18, 2024
Maintainer

@deansheather can we adjust this?

0 replies

matifali · 2024-12-18T07:10:41Z

matifali
Dec 18, 2024
Maintainer

Hi, @groundsada could you share the logs and errors you get when it times out? Are you using the builtin example Kubernetes template that uses a kubernets_deployment?
If yes you can increase it by setting timeouts.

0 replies

groundsada · 2024-12-18T15:29:16Z

groundsada
Dec 18, 2024
Author

HI @matifali

These are the Build Logs:

Terraform 1.9.2
coder_agent.main: Plan to create
coder_app.jupyter: Plan to create
coder_app.code-server: Plan to create
coder_app.cursor: Plan to create
kubernetes_persistent_volume_claim.home: Plan to create
kubernetes_pod.main[0]: Plan to create
coder_agent.main: Creating...
coder_agent.main: Creation complete after 0s [id=db70a6a2-ccbd-4ae9-bca2-16eb8bcbe265]
coder_app.cursor: Creating...
coder_app.jupyter: Creating...
coder_app.code-server: Creating...
coder_app.cursor: Creation complete after 0s [id=bdf8bc9e-b116-4212-ac1f-10f2556f05aa]
kubernetes_persistent_volume_claim.home: Creating...
coder_app.jupyter: Creation complete after 0s [id=4aa30645-118a-4e80-b92f-de7ea212d790]
coder_app.code-server: Creation complete after 0s [id=ffdcec70-a401-45f1-ab7e-8b70e0f37c5b]
kubernetes_persistent_volume_claim.home: Creation complete after 1s [id=coder/coder-mfsada-pytorch1-home]
kubernetes_pod.main[0]: Creating...
kubernetes_pod.main[0]: Still creating... [10s elapsed]
kubernetes_pod.main[0]: Still creating... [20s elapsed]
kubernetes_pod.main[0]: Still creating... [30s elapsed]
kubernetes_pod.main[0]: Still creating... [40s elapsed]
kubernetes_pod.main[0]: Still creating... [50s elapsed]
kubernetes_pod.main[0]: Still creating... [1m0s elapsed]
kubernetes_pod.main[0]: Still creating... [1m10s elapsed]
kubernetes_pod.main[0]: Still creating... [1m20s elapsed]
kubernetes_pod.main[0]: Still creating... [1m30s elapsed]
kubernetes_pod.main[0]: Still creating... [1m40s elapsed]
kubernetes_pod.main[0]: Still creating... [1m50s elapsed]
kubernetes_pod.main[0]: Still creating... [2m0s elapsed]
kubernetes_pod.main[0]: Still creating... [2m10s elapsed]
kubernetes_pod.main[0]: Still creating... [2m20s elapsed]
kubernetes_pod.main[0]: Still creating... [2m30s elapsed]
kubernetes_pod.main[0]: Still creating... [2m40s elapsed]
kubernetes_pod.main[0]: Still creating... [2m50s elapsed]
kubernetes_pod.main[0]: Still creating... [3m0s elapsed]
kubernetes_pod.main[0]: Still creating... [3m10s elapsed]
kubernetes_pod.main[0]: Still creating... [3m20s elapsed]
kubernetes_pod.main[0]: Still creating... [3m30s elapsed]
kubernetes_pod.main[0]: Still creating... [3m40s elapsed]
kubernetes_pod.main[0]: Still creating... [3m50s elapsed]
kubernetes_pod.main[0]: Still creating... [4m0s elapsed]
kubernetes_pod.main[0]: Still creating... [4m10s elapsed]
kubernetes_pod.main[0]: Still creating... [4m20s elapsed]
kubernetes_pod.main[0]: Still creating... [4m30s elapsed]
kubernetes_pod.main[0]: Still creating... [4m40s elapsed]
kubernetes_pod.main[0]: Still creating... [4m50s elapsed]
kubernetes_pod.main[0]: Still creating... [5m0s elapsed]
kubernetes_pod.main[0]: Creation errored after 5m0s
Error: client rate limiter Wait returned an error: context deadline exceeded
on main.tf line 215, in resource "kubernetes_pod" "main":
  215: resource "kubernetes_pod" "main" {

This happens when pulling a big image takes more than 5 minutes. The pod is actually running now without Coder knowing.

Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Warning  FailedScheduling        8m51s (x2 over 8m54s)  default-scheduler        0/422 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/422 nodes are available: 422 Preemption is not helpful for scheduling..
  Normal   Scheduled               8m48s                  default-scheduler        Successfully assigned coder/coder-mfsada-pytorch1 to k8s-gpu-1.ucsc.edu
  Normal   SuccessfulAttachVolume  8m48s                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-faae7a31-dac4-4bb8-9721-6453dbfa69a0"
  Normal   Pulling                 8m37s                  kubelet                  Pulling image "gitlab-registry.nrp-nautilus.io/nrp/coder-images/pytorch"
  Normal   Pulled                  3m19s                  kubelet                  Successfully pulled image "gitlab-registry.nrp-nautilus.io/nrp/coder-images/pytorch" in 5m18.33s (5m18.33s including waiting)
  Normal   Created                 3m18s                  kubelet                  Created container dev
  Normal   Started                 3m18s                  kubelet                  Started container dev

0 replies

groundsada · 2024-12-18T15:31:38Z

groundsada
Dec 18, 2024
Author

@matifali My template is built from an example that uses a kubernetes pod and PVC, this is the pod snippet:

resource "kubernetes_pod" "main" {
   count = data.coder_workspace.me.start_count
  metadata {
    name      = "coder-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}"
    namespace = var.namespace
  }

  spec {
    security_context {
      run_as_user = "1000"
      fs_group    = "1000"
      fs_group_change_policy = "OnRootMismatch"
    }
    container {
      name              = "dev"
      image             = "gitlab-registry.nrp-nautilus.io/nrp/coder-images/pytorch"
      image_pull_policy = "IfNotPresent"
      command           = ["sh", "-c", coder_agent.main.init_script]
      security_context {
        run_as_user = "1000"
      }
      env {
        name  = "CODER_AGENT_TOKEN"
        value = coder_agent.main.token
      }
      resources {
        requests = {
          "cpu"    = "${data.coder_parameter.cpu.value}"
          "memory" = "${data.coder_parameter.memory.value}Gi"
          "${data.coder_parameter.gpu_type.value}" = "${data.coder_parameter.gpu.value}"
        }
        limits = {
          "cpu"    = "${data.coder_parameter.cpu.value}"
          "memory" = "${data.coder_parameter.memory.value}Gi"
          "${data.coder_parameter.gpu_type.value}" = "${data.coder_parameter.gpu.value}"
        }
      }

      volume_mount {
        mount_path = "/home/coder"
        name       = "home"
        read_only  = false
      }
      volume_mount {
        mount_path = "/dev/shm"
        name       = "dshm"
      }
    }

    volume {
      name = "home"
      persistent_volume_claim {
        claim_name = kubernetes_persistent_volume_claim.home.metadata.0.name
        read_only  = false
      }
    }
    volume {
      name = "dshm"
      empty_dir {
        medium = "Memory"
        size_limit = "2Gi" # Adjust size as needed
      }
    }

  }
}

4 replies

matifali Dec 18, 2024
Maintainer

Could you try using timeouts then?

Answer selected by matifali

groundsada Jan 6, 2025
Author

Hi @matifali
Using timeouts for create for 5m did not change the behavior for large images. The pod is still on ConatinerCreating after 5 minutes have passed and Coder still shows failure.

matifali Jan 6, 2025
Maintainer

What is the value of the timeout you set? Try setting up to a higher value like 30m

groundsada Jan 6, 2025
Author

Yes! This is exactly what I needed for my deployment! Thank you @matifali

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Increase Timeout from 5 Minutes (K8s pods orphaned) #15909

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Increase Timeout from 5 Minutes (K8s pods orphaned) #15909

groundsada Dec 18, 2024

Replies: 4 comments · 4 replies

matifali Dec 18, 2024 Maintainer

matifali Dec 18, 2024 Maintainer

groundsada Dec 18, 2024 Author

groundsada Dec 18, 2024 Author

matifali Dec 18, 2024 Maintainer

groundsada Jan 6, 2025 Author

matifali Jan 6, 2025 Maintainer

groundsada Jan 6, 2025 Author

groundsada
Dec 18, 2024

Replies: 4 comments 4 replies

matifali
Dec 18, 2024
Maintainer

matifali
Dec 18, 2024
Maintainer

groundsada
Dec 18, 2024
Author

groundsada
Dec 18, 2024
Author

matifali Dec 18, 2024
Maintainer

groundsada Jan 6, 2025
Author

matifali Jan 6, 2025
Maintainer

groundsada Jan 6, 2025
Author