From dcddb3adbae8a1f7113d4fe639b553a001801550 Mon Sep 17 00:00:00 2001 From: Mark Milligan Date: Sun, 14 Aug 2022 15:24:43 -0500 Subject: [PATCH 1/5] chore: refactor aws eks install page - reduce nodes, make ubuntu default --- setup/kubernetes/aws.md | 271 +++++++++++++++------------------------- 1 file changed, 101 insertions(+), 170 deletions(-) diff --git a/setup/kubernetes/aws.md b/setup/kubernetes/aws.md index 0ffb3fab9..8ebdd5329 100644 --- a/setup/kubernetes/aws.md +++ b/setup/kubernetes/aws.md @@ -5,7 +5,7 @@ description: --- This deployment guide shows you how to set up an Amazon Elastic Kubernetes -Engine cluster on which Coder can deploy. +Engine "EKS" cluster on which Coder can deploy. ## Prerequisites @@ -21,195 +21,74 @@ machine: to fast-track this process - [eksctl command-line utility](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) -## Node Considerations - -The node type and size that you select impact how you use Coder. When choosing, -be sure to account for the number of developers you expect to use Coder, as well -as the resources they need to run their workspaces. See our guide on on -[compute resources](../../guides/admin/resources.md) for additional information. - -If you expect to provision GPUs to your Coder workspaces, you **must** use an -EC2 instance from AWS' -[accelerated computing instance family](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/accelerated-computing-instances.html). - -> GPUs are not supported in workspaces deployed as -> [container-based virtual machines (CVMs)](../../workspaces/cvms.md) unless -> you're running Coder in a bare-metal Kubernetes environment. - -## Preliminary steps - -Before you can create a cluster, you'll need to perform the following to set up -and configure your AWS account. - -1. Go to AWS' [EC2 console](https://console.aws.amazon.com/ec2/); this should - take you to the EC2 page for the AWS region in which you're working (if not, - change to the correct region using the dropdown in the top-right of the page) -1. In the **Resources** section in the middle of the page, click **Elastic - IPs**. -1. Choose either an Elastic IP address you want to use or click **Allocate - Elastic IP address**. Choose **Amazon's pool of IPv4 addresses** and click - **Allocate**. -1. Return to the EC2 Dashboard. -1. In the **Resources** section in the middle of the page, click **Key Pairs**. -1. Click **Create key pair** (alternatively, if you already have a local SSH key - you'd like to use, you can click the Actions dropdown and import your key) -1. Provide a **name** for your key pair and select **pem** as your **file - format**. Click **Create key pair**. -1. You'll automatically download the keypair; save it to a known directory on - your local machine (we recommend keeping the default name, which will match - the name you provided to AWS). -1. Now that you have the `.pem` file, extract the public key portion of the - keypair so that you can use it with the eksctl CLI in later steps: +See [Preliminary Steps](#preliminary-steps) and [Node Considerations](#node-considerations) first to familiarize yourself with steps and items before creating a cluster. - ```sh - ssh-keygen -y -f .pem >> .pub - ``` +## Step 1: Create an EKS cluster - **Note**: if you run into a bad permissions error, run `sudo` before the - command above. - -When done, you should have a .pem and .pub file for the same keypair you -downloaded from AWS. +While flags can be passed to `eksctl create cluster`, the following example uses +a configuration yaml file to define an EKS cluster. -## Step 1: Spin up a K8s cluster - -To make subsequent steps easier, start by creating environment variables for the -cluster name, +> The cluster name, [region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions), -and SSH key path: - -```console -CLUSTER_NAME="YOUR_CLUSTER_NAME" -SSH_KEY_PATH=".pub" -REGION="YOUR_REGION" +and SSH key path will be specific to your installation so potentially change +them in the yaml file + + +```yaml +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig + +metadata: + name: trial-cluster + region: us-east-1 + +managedNodeGroups: + - name: managed-ng-1 + spot: true + instanceType: t2.medium + amiFamily: Ubuntu2004 + desiredCapacity: 1 + minSize: 1 + maxSize: 2 + volumeSize: 50 + ssh: + allow: true + publicKeyPath: ~/.ssh/id_rsa.pub ``` -The following will spin up a Kubernetes cluster using `eksctl` (be sure to -update the parameters as necessary, especially the version number): +> If your developers require Docker commands like `docker build`, `docker run`, and `docker-compose` as +> part of their development flow, then [container-based virtual machines +> (CVMs)](../../workspaces/cvms.md) are required. `instantType` of `Ubuntu2004` +> is preferred since `AmazonLinux2` does not support caching and the shiftfs +> kernel module in CVM settings. -```console - - eksctl create cluster \ - --name "$CLUSTER_NAME" \ - --version \ - --region "$REGION" \ - --nodegroup-name standard-workers \ - --node-type t3.medium \ - --nodes 2 \ - --nodes-min 2 \ - --nodes-max 8 \ - --ssh-access \ - --ssh-public-key "$SSH_KEY_PATH" \ - --managed -``` - -Please note that the sample script creates a `t3.medium` instance; depending on +This example uses `t2.medium` instance with 2 nodes which is meant for a small deployment to trial Coder. Depending on your needs, you can choose a -[larger size](https://aws.amazon.com/ec2/instance-types/t3/) instead. See +[larger size](https://aws.amazon.com/ec2/instance-types/) instead. See [requirements](../requirements.md) for help estimating your cluster size. When your cluster is ready, you should see the following message: ```console -EKS cluster "YOUR_CLUSTER_NAME" in "YOUR_REGION" region is ready +EKS cluster "YOUR CLUSTER NAME" in "YOUR REGION" region is ready ``` -This process may take ~15-30 minutes to complete. - -## Step 2: Adjust the K8s storage class - -Once you've created the cluster, adjust the default Kubernetes storage class to -support immediate volume binding. - -1. Make sure that you're pointed to the correct context: - - ```console - kubectl config current-context - ``` - -1. If you're pointed to the correct context, delete the gp2 storage class: - - ```console - kubectl delete sc gp2 - ``` - -1. Recreate the gp2 storage class with the `volumeBindingMode` set to - `Immediate`: - - ```console - cat < See the -> [Kubernetes docs](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) -> for information on choosing the right parameter for `volumeBindingMode`; Coder -> accepts both `Immediate` and `WaitForFirstConsumer`. - -### Modifying your cluster to support CVMs - -To create clusters allowing you to -[enable container-based virtual machines (CVMs)](../../admin/workspace-management/cvms.md) -as a workspace deployment option, you'll need to -[create a nodegroup](https://eksctl.io/usage/eks-managed-nodes/#creating-managed-nodegroups). - -1. Define your config file (we've named the file `coder-node.yaml`, but you can - call it whatever you'd like): - - ```yaml - apiVersion: eksctl.io/v1alpha5 - kind: ClusterConfig - - metadata: - version: "" - name: - region: - - managedNodeGroups: - - name: coder-node-group - amiFamily: Ubuntu2004 # AmazonLinux2 is also a supported option - # Custom EKS-compatible AMIs can be used instead of amiFamily - # ami: - instanceType: - minSize: 1 - maxSize: 2 - desiredCapacity: 1 - # Uncomment "overrideBootstrapCommand" if you are using a custom AMI - # overrideBootstrapCommand: | - # #!/bin/bash -xe - # sudo /etc/eks/bootstrap.sh - ``` +This process may take ~15-30 minutes to complete since it is creating EC2 instance(s) aka node(s), node pool, a VPC, NAT Gateway, network interface, security group, elastic IP, EKS cluster, namespaces and pods. -> See -> [the list of EKS-compatible Ubuntu AMIs](https://cloud-images.ubuntu.com/docs/aws/eks/) -> and info on -> [Latest & Custom AMI support](https://eksctl.io/usage/custom-ami-support/). +> EKS creates a `volumeBindingMode` of `WaitForFirstConsumer`. See the +> [Kubernetes +> docs](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) +> for more information. Coder accepts both `Immediate` and +> `WaitForFirstConsumer`. -1. Create your nodegroup (be sure to provide the correct file name): - ```console - eksctl create nodegroup --config-file=coder-node.yaml - ``` - -## Step 3: (Optional) Install Calico onto your cluster +## Step 2: (Optional) Install Calico onto your cluster AWS uses [Calico](https://docs.amazonaws.cn/en_us/eks/latest/userguide/calico.html) to -implement network segmentation and tenant isolation. We strongly recommend -executing the following steps; please see -[Network Policies](../requirements.md#network-policies) for more information. +implement network segmentation and tenant isolation. For production deployments, +we recommend Calico to enforce workspace pod isolation; please see [Network +Policies](../requirements.md#network-policies) for more information. 1. Apply the Calico manifest to your cluster: @@ -232,6 +111,58 @@ executing the following steps; please see calico-node 3 3 3 3 ... ``` +## Cleanup | Delete EKS cluster + +To delete the EKS cluster including any installation of Coder, substitute your cluster name and zone in the following `eksctl` command. This will take several minutes and can be monitored in the CloudFormation stack. + +```console +eksctl delete cluster --region=us-east-1 --name=trial-cluster +``` + +## Preliminary steps + +Before you can create a cluster, you'll need to perform the following to set up +and configure your AWS account. + +1. Go to AWS' [EC2 console](https://console.aws.amazon.com/ec2/); this should + take you to the EC2 page for the AWS region in which you're working (if not, + change to the correct region using the dropdown in the top-right of the page) +1. In the **Resources** section in the middle of the page, click **Key Pairs**. +1. Click **Create key pair** (alternatively, if you already have a local SSH key + you'd like to use, you can click the Actions dropdown and import your key) +1. Provide a **name** for your key pair and select **pem** as your **file + format**. Click **Create key pair**. +1. You'll automatically download the keypair; save it to a known directory on + your local machine (we recommend keeping the default name, which will match + the name you provided to AWS). +1. Now that you have the `.pem` file, extract the public key portion of the + keypair so that you can use it with the eksctl CLI in later steps: + + ```sh + ssh-keygen -y -f .pem >> .pub + ``` + + **Note**: if you run into a bad permissions error, run `sudo` before the + command above. + +When done, you should have a .pem and .pub file for the same keypair you +downloaded from AWS. + +## Node Considerations + +The node type and size that you select impact how you use Coder. When choosing, +be sure to account for the number of developers you expect to use Coder, as well +as the resources they need to run their workspaces. See our guide on on +[compute resources](../../guides/admin/resources.md) for additional information. + +If you expect to provision GPUs to your Coder workspaces, you **must** use an +EC2 instance from AWS' +[accelerated computing instance family](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/accelerated-computing-instances.html). + +> GPUs are not supported in workspaces deployed as +> [container-based virtual machines (CVMs)](../../workspaces/cvms.md) unless +> you're running Coder in a bare-metal Kubernetes environment. + ## Access control EKS allows you to create and manage user permissions using IAM identity @@ -255,4 +186,4 @@ provider](../../admin/workspace-providers/deployment/index.md). To access Coder through a secure domain, review our guides on configuring and using [TLS certificates](../../guides/tls-certificates/index.md). -Once complete, see our page on [installation](../installation.md). +Once complete, see our page on [Coder installation](../installation.md). From c32afdc85a7457765a3f3e964cb841db50abcf5f Mon Sep 17 00:00:00 2001 From: Mark Milligan Date: Sun, 14 Aug 2022 15:31:24 -0500 Subject: [PATCH 2/5] chore: add eksctl create cluster command --- setup/kubernetes/aws.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/setup/kubernetes/aws.md b/setup/kubernetes/aws.md index 8ebdd5329..fb369f0c0 100644 --- a/setup/kubernetes/aws.md +++ b/setup/kubernetes/aws.md @@ -56,6 +56,10 @@ managedNodeGroups: publicKeyPath: ~/.ssh/id_rsa.pub ``` +```console +eksctl create cluster -f cluster.yaml +``` + > If your developers require Docker commands like `docker build`, `docker run`, and `docker-compose` as > part of their development flow, then [container-based virtual machines > (CVMs)](../../workspaces/cvms.md) are required. `instantType` of `Ubuntu2004` From 6eba939665c7e7432a8898e35001e45ca0784fb4 Mon Sep 17 00:00:00 2001 From: Mark Milligan Date: Sun, 14 Aug 2022 16:43:01 -0500 Subject: [PATCH 3/5] chore: remove spot=true - was used for my testing --- setup/kubernetes/aws.md | 1 - 1 file changed, 1 deletion(-) diff --git a/setup/kubernetes/aws.md b/setup/kubernetes/aws.md index fb369f0c0..505223e6b 100644 --- a/setup/kubernetes/aws.md +++ b/setup/kubernetes/aws.md @@ -44,7 +44,6 @@ metadata: managedNodeGroups: - name: managed-ng-1 - spot: true instanceType: t2.medium amiFamily: Ubuntu2004 desiredCapacity: 1 From 330cd3ca46fcbdd4b41005aaa992d03c99084c50 Mon Sep 17 00:00:00 2001 From: Eric Paulsen Date: Mon, 15 Aug 2022 12:08:19 -0400 Subject: [PATCH 4/5] refactor: EKS install guide --- setup/kubernetes/aws.md | 121 ++++++++++------------------------------ 1 file changed, 28 insertions(+), 93 deletions(-) diff --git a/setup/kubernetes/aws.md b/setup/kubernetes/aws.md index 505223e6b..05632dcdb 100644 --- a/setup/kubernetes/aws.md +++ b/setup/kubernetes/aws.md @@ -5,7 +5,7 @@ description: --- This deployment guide shows you how to set up an Amazon Elastic Kubernetes -Engine "EKS" cluster on which Coder can deploy. +Engine (EKS) cluster on which Coder can deploy. ## Prerequisites @@ -21,25 +21,22 @@ machine: to fast-track this process - [eksctl command-line utility](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) -See [Preliminary Steps](#preliminary-steps) and [Node Considerations](#node-considerations) first to familiarize yourself with steps and items before creating a cluster. - ## Step 1: Create an EKS cluster While flags can be passed to `eksctl create cluster`, the following example uses -a configuration yaml file to define an EKS cluster. +an [`eksctl` configuration file](https://eksctl.io/usage/schema/) to define the +EKS cluster. > The cluster name, -[region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions), -and SSH key path will be specific to your installation so potentially change -them in the yaml file - +> [region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones>.html#concepts-regions), +> and SSH key path will be specific to your installation. ```yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: - name: trial-cluster + name: coder-trial-cluster region: us-east-1 managedNodeGroups: @@ -49,26 +46,31 @@ managedNodeGroups: desiredCapacity: 1 minSize: 1 maxSize: 2 - volumeSize: 50 + volumeSize: 100 ssh: allow: true publicKeyPath: ~/.ssh/id_rsa.pub ``` +This example uses `t2.medium` instance with 2 nodes which is meant for a small +trial deployment. Depending on your needs, you can choose a +[larger size](https://aws.amazon.com/ec2/instance-types/) instead. See our +documentation on [resources](../../guides/admin/resources.md) and +[requirements](../requirements.md) for help estimating your cluster size. + +Once the file is ready, run the following command to create the cluster: + ```console eksctl create cluster -f cluster.yaml ``` -> If your developers require Docker commands like `docker build`, `docker run`, and `docker-compose` as -> part of their development flow, then [container-based virtual machines -> (CVMs)](../../workspaces/cvms.md) are required. `instantType` of `Ubuntu2004` -> is preferred since `AmazonLinux2` does not support caching and the shiftfs -> kernel module in CVM settings. +This process may take ~15-30 minutes to complete since it is creating EC2 +instance(s) aka node(s), node pool, a VPC, NAT Gateway, network interface, +security group, elastic IP, EKS cluster, namespaces and pods. -This example uses `t2.medium` instance with 2 nodes which is meant for a small deployment to trial Coder. Depending on -your needs, you can choose a -[larger size](https://aws.amazon.com/ec2/instance-types/) instead. See -[requirements](../requirements.md) for help estimating your cluster size. +> By default, EKS creates a `volumeBindingMode` of `WaitForFirstConsumer`. See the +> [Kubernetes docs](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) +> for more information on this mode. Coder accepts both `Immediate` and `WaitForFirstConsumer`. When your cluster is ready, you should see the following message: @@ -76,22 +78,12 @@ When your cluster is ready, you should see the following message: EKS cluster "YOUR CLUSTER NAME" in "YOUR REGION" region is ready ``` -This process may take ~15-30 minutes to complete since it is creating EC2 instance(s) aka node(s), node pool, a VPC, NAT Gateway, network interface, security group, elastic IP, EKS cluster, namespaces and pods. - -> EKS creates a `volumeBindingMode` of `WaitForFirstConsumer`. See the -> [Kubernetes -> docs](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) -> for more information. Coder accepts both `Immediate` and -> `WaitForFirstConsumer`. - - ## Step 2: (Optional) Install Calico onto your cluster -AWS uses -[Calico](https://docs.amazonaws.cn/en_us/eks/latest/userguide/calico.html) to -implement network segmentation and tenant isolation. For production deployments, -we recommend Calico to enforce workspace pod isolation; please see [Network -Policies](../requirements.md#network-policies) for more information. +AWS uses [Calico](https://docs.amazonaws.cn/en_us/eks/latest/userguide/calico.html) +to implement network segmentation and tenant isolation. For production deployments, +we recommend Calico to enforce workspace pod isolation; please see [Network Policies](../requirements.md#network-policies) +for more information. 1. Apply the Calico manifest to your cluster: @@ -116,71 +108,14 @@ Policies](../requirements.md#network-policies) for more information. ## Cleanup | Delete EKS cluster -To delete the EKS cluster including any installation of Coder, substitute your cluster name and zone in the following `eksctl` command. This will take several minutes and can be monitored in the CloudFormation stack. +To delete the EKS cluster including any installation of Coder, substitute your +cluster name and zone in the following `eksctl` command. This will take several +minutes and can be monitored in the CloudFormation stack. ```console eksctl delete cluster --region=us-east-1 --name=trial-cluster ``` -## Preliminary steps - -Before you can create a cluster, you'll need to perform the following to set up -and configure your AWS account. - -1. Go to AWS' [EC2 console](https://console.aws.amazon.com/ec2/); this should - take you to the EC2 page for the AWS region in which you're working (if not, - change to the correct region using the dropdown in the top-right of the page) -1. In the **Resources** section in the middle of the page, click **Key Pairs**. -1. Click **Create key pair** (alternatively, if you already have a local SSH key - you'd like to use, you can click the Actions dropdown and import your key) -1. Provide a **name** for your key pair and select **pem** as your **file - format**. Click **Create key pair**. -1. You'll automatically download the keypair; save it to a known directory on - your local machine (we recommend keeping the default name, which will match - the name you provided to AWS). -1. Now that you have the `.pem` file, extract the public key portion of the - keypair so that you can use it with the eksctl CLI in later steps: - - ```sh - ssh-keygen -y -f .pem >> .pub - ``` - - **Note**: if you run into a bad permissions error, run `sudo` before the - command above. - -When done, you should have a .pem and .pub file for the same keypair you -downloaded from AWS. - -## Node Considerations - -The node type and size that you select impact how you use Coder. When choosing, -be sure to account for the number of developers you expect to use Coder, as well -as the resources they need to run their workspaces. See our guide on on -[compute resources](../../guides/admin/resources.md) for additional information. - -If you expect to provision GPUs to your Coder workspaces, you **must** use an -EC2 instance from AWS' -[accelerated computing instance family](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/accelerated-computing-instances.html). - -> GPUs are not supported in workspaces deployed as -> [container-based virtual machines (CVMs)](../../workspaces/cvms.md) unless -> you're running Coder in a bare-metal Kubernetes environment. - -## Access control - -EKS allows you to create and manage user permissions using IAM identity -providers (IdPs). EKS also supports user authentication via OpenID Connect -(OIDC) identity providers. - -Using IAM with Kubernetes' native Role-Based Access Control (RBAC) allows you to -grant access to your EKS cluster using existing IDPs and fine-tune permissions -with RBAC. - -For more information, see: - -- [AWS identity providers and federation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers.html) -- [Kubernetes RBAC authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) - ## Next steps If you have already installed Coder, you can add this cluster as a [workspace From c9d0f467f6899d85a22c753878717db4955adbfb Mon Sep 17 00:00:00 2001 From: Eric Paulsen Date: Mon, 15 Aug 2022 13:06:11 -0400 Subject: [PATCH 5/5] re-add: CVM note --- setup/kubernetes/aws.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/setup/kubernetes/aws.md b/setup/kubernetes/aws.md index 05632dcdb..8331adc80 100644 --- a/setup/kubernetes/aws.md +++ b/setup/kubernetes/aws.md @@ -58,6 +58,13 @@ trial deployment. Depending on your needs, you can choose a documentation on [resources](../../guides/admin/resources.md) and [requirements](../requirements.md) for help estimating your cluster size. + > If your developers require Docker commands like `docker build`, `docker run`, + > and `docker-compose` as part of their development flow, then + > [container-based virtual machines (CVMs)](../../workspaces/cvms.md) are + > required. In this case, we recommend using the `Ubuntu2004` AMI family, as + > the `AmazonLinux2` AMI family does not meet the requirements + > for [cached CVMs](../../workspace-management/cvms/management#caching). + Once the file is ready, run the following command to create the cluster: ```console