Skip to content

User-local provisioner daemons #909

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
kconley-sq opened this issue Apr 7, 2022 · 10 comments
Closed

User-local provisioner daemons #909

kconley-sq opened this issue Apr 7, 2022 · 10 comments
Assignees
@kylecarbs
Labels
api Area: HTTP API

Comments

@kconley-sq
Copy link
Contributor

Our team's developers need to access their local USB devices in their Coder workspaces via USB/IP, and to minimize the latency between a USB/IP server running on a developer's local machine (which serves the USB hardware they have connected to their machine) and the USB/IP client running in a Coder workspace (which attaches that USB hardware remotely over the network), we'd like our developers to be able to provision Coder workspaces to run on their local machines (e.g. using the Docker Terraform provider).

I was describing this use case to @kylecarbs on Slack, and he mentioned:

Coder v2 has the concept of a provisioner daemon. It accepts provision operations (like workspace start/stop) and performs them. Developers could run a provisioner daemon on their local machine, and then Terraform would execute against it. This would enable Terraform to run locally, so you could use unix:///var/run/docker.sock. Our architecture certainly enables this.

We'll have to work with y'all to get the UX nice, but we can certainly provide a flow like:

  1. Download the Coder CLI.
  2. Run "coder provisioner" in the background.
  3. Have the workspace provisioned on your local machine, using that provisioner daemon.

We'll have to add some targeting to provisionerd, but that should be simple enough!

That sounds exactly like what we were hoping for, so I've created this GitHub issue to formally request this feature in Coder. 😃

Building on what @kylecarbs suggested above, I think enabling something like the following would meet our requirements:

  1. A developer can run a Coder CLI command (e.g. coder provisioner) to launch a local provisioner that is associated/authenticated with coderd using the credentials/URL the developer previously provided via coder login.
  2. A developer can specify that workspace operations (e.g. creation, stopping, etc.) should use a particular provisioner daemon so that they can ensure that a workspace will be created/provisioned/stopped on their local machine as opposed to other machines that would not immediately have the ability to provision/control Terraform resources on the developer's local machine.
    • For the CLI, maybe this could be specified via an option when running coder workspace create like --provisioner-daemon <PROVISIONER_DAEMON_ID> where a developer specifies the name/reference to the provisioner daemon of interest via <PROVISIONER_DAEMON_ID>?
    • For the web UI, maybe this could be a drop-down on the workspace creation webpage from which the user could select a provisioner daemon from a list of provisioner daemons that they are authorized to use (see below)?
  3. A developer can restrict other Coder users from creating workspaces using a provisioner daemon they are running on their local machine. I think this would help prevent one developer from accidentally (or perhaps maliciously) creating a workspace on a different developer's machine as well as avoid having too many options for selecting a provisioner daemon in the new workspace creation web UI (see above about the proposed drop-down option) when many developers maintain their own provisioner daemons.
    • I think something that would work well for our team would be if the default behavior of Coder would be to give a developer the ability to create workspaces using a provisioner daemon they've spawned themselves but restrict other users from utilizing it unless the provisioner daemon has been associated/registered to be shared throughout a Coder organization. And maybe that association/registration could be performed by the developer via a CLI command or web UI page?

What do you think?

cc @sunilvenkatraman
@kylecarbs kylecarbs self-assigned this Apr 12, 2022
@kylecarbs
Copy link
Member

@kconley-sq I was able to get this working. It's not scoped to a specific user right now, but we are able to use the Docker Terraform Provider to spin up a workspace. This is a basic template that lets you specify an image. Pretty neat!

terraform {
  required_providers {
    coder = {
      source  = "coder/coder"
      version = "~> 0.3.1"
    }
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 2.16.0"
    }
  }
}

provider "coder" {
  url = "http://host.docker.internal:3000/"
}

provider "docker" {
  host = "unix:///var/run/docker.sock"
}

resource "coder_agent" "dev" {
  arch = "amd64"
  os   = "linux"
}

resource "docker_image" "go" {
  name = "mcr.microsoft.com/vscode/devcontainers/go:1"
}

resource "docker_container" "go" {
  image      = docker_image.go.latest
  name       = "go"
  entrypoint = ["sh", "-c", coder_agent.dev.init_script]
  user       = "vscode"
  host {
    host = "host.docker.internal"
    ip   = "host-gateway"
  }
  env = [
    "CODER_TOKEN=${coder_agent.dev.token}"
  ]
}

@bpmct
Copy link
Member

bpmct commented May 5, 2022

@arivictor would something like this work for your use case of local machines?

@tjcran
Copy link

tjcran commented May 16, 2022

I am adding this one into Community MVP milestone. It unblocks specific use cases we've heard from multiple people now.

@tjcran tjcran added this to the Community MVP milestone May 16, 2022
@tjcran tjcran mentioned this issue May 16, 2022
Closed
@misskniss
Copy link

Please add your planning poker estimate with ZenHub @kylecarbs

@misskniss misskniss modified the milestones: Enterprise MVP , Post-MVP Jun 3, 2022
@misskniss misskniss removed this from the Post-MVP milestone Jul 22, 2022
@tjcran tjcran mentioned this issue Jul 26, 2022
Closed
25 tasks
@dwahler
Copy link
Contributor

dwahler commented Jul 28, 2022

It sounds like there are a few prerequisites to being able to implement this as more than just a demo:

First of all, we need to decide on a notion of identity for provisioners, so that we can distinguish between different instances of the Terraform provisioner. Simplest thing is probably just to assign each of them a UUID, but we might also want to consider namespacing them, so that you'd have e.g. global and dwahler/local.

Secondly, we need to think about security (authn/authz). #44 covers authentication between coderd and provisionerd. For authorization, @kconley-sq's proposed defaults sound great, but I can think of some other potential things to consider:

  • Should every user be allowed to run local provisioners?
  • If they have a local provisioner, does that automatically imply the ability to create their own templates?
  • Do we need to restrict which templates can run on which provisioners, to avoid leaking sensitive template parameters?
  • Should we require some kind of local user confirmation before running arbitrary Terraform code that we receive from the remote coderd?

And then there are lifecycle issues. It probably doesn't make sense to delete a provisioner instance if there are existing workspaces associated with it. But if the provisioner goes away unexpectedly (e.g. your laptop dies) then we probably need a way to clean up its state.

@dwahler dwahler mentioned this issue Jul 28, 2022
Closed
@misskniss misskniss assigned dwahler and unassigned kylecarbs Jul 29, 2022
@kconley-sq kconley-sq mentioned this issue Aug 29, 2022
Closed
@github-actions
Copy link

This issue is becoming stale. In order to keep the tracker readable and actionable, I'm going close to this issue in 7 days if there isn't more activity.

@github-actions github-actions bot added the stale This issue is like stale bread. label Oct 22, 2022
@bpmct bpmct assigned kylecarbs and unassigned dwahler Oct 26, 2022
@github-actions github-actions bot removed the stale This issue is like stale bread. label Oct 27, 2022
@kylecarbs
Copy link
Member

kylecarbs commented Nov 7, 2022 •
edited
Loading

#4935

@kylecarbs
Copy link
Member

Provisioner daemons and template versions will gain the concept of tags. Tags must match for a provision job to execute on the daemon.

Tags will be defined in the frontmatter of a template's README:

---
tags:
- scope: organization
- environment: staging
---

# Title

Some additional information...

When a provisioner job is queued (like template creation or workspace build), it will be clear to the user that the job is pending on a set of tags. If provisioner: user is specified, we'll make a unique exception in the UI directing the user to start a provisioner daemon. All users can create provisioner daemons, but they are useless unless templates target them.

# Registers and starts a new provisioner daemon with a set of tags
$ coder provisionerd start --tag environment=staging --tag scope=organization

The scope tag specifies whether the provisioner daemon is registered for the organization or the user. If not provided, the default value is user. If the user doesn't have permission to create an organization-wide provisioner daemon, the request to start will fail.

The following tags will be specified by default but can be overridden by the user:

  • arch: Specifies the architecture of the host system.
  • os: Specifies the operating system of the host system.

@kylecarbs kylecarbs mentioned this issue Nov 16, 2022
Merged
@kylecarbs
Copy link
Member

This is done!

@kconley-sq kconley-sq mentioned this issue Aug 25, 2023
Closed
@matifali
Copy link
Member

matifali commented Jan 4, 2024

@bpmct @kylecarbs, We currently do not have the ability to run a workspace on the developer's local docker daemon (as requested originally in the issue); I think we should reopen this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kylecarbs kylecarbs

Labels
api Area: HTTP API
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@dwahler @misskniss @tjcran @kylecarbs @matifali @bpmct @kconley-sq