Skip to content

ci: use a wildcard subdomain for PR deployments #8801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Aug 1, 2023
Merged

ci: use a wildcard subdomain for PR deployments #8801

Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
0f00e25
wip
matifali Jul 27, 2023
0093621
fmt
matifali Jul 30, 2023
f3f6669
fix binary path
matifali Jul 30, 2023
198c429
fix
matifali Jul 30, 2023
6bd5c80
fix info in slack notification
matifali Jul 30, 2023
27f37ce
run in subshell
matifali Jul 30, 2023
41c9d70
fix credentials output
matifali Jul 30, 2023
e76c683
change username
matifali Jul 30, 2023
2e74a83
Merge branch 'main' into use-wildcard-domain
matifali Jul 30, 2023
67858e4
Update pr-cleanup.yaml
matifali Jul 30, 2023
e526434
Typo
matifali Jul 30, 2023
1fbf9ff
use subshells
matifali Jul 30, 2023
5ed660e
allow custom experiments
matifali Jul 30, 2023
b1e9663
handle extra experiments
matifali Jul 30, 2023
382ecca
handle concurrency
matifali Jul 30, 2023
7ad2857
update
matifali Jul 30, 2023
81cf4c3
fixup
matifali Jul 30, 2023
48d8871
add experiments flag
matifali Jul 30, 2023
af9b588
wip
matifali Jul 30, 2023
bc881dc
use coder ingress
matifali Jul 31, 2023
dac8997
fix EOF
matifali Jul 31, 2023
c9daeaf
fix lint
matifali Jul 31, 2023
8192128
double quotes
matifali Jul 31, 2023
3e6d936
store secret in a shared namespace
matifali Aug 1, 2023
5ade8b7
sanitize secret
matifali Aug 1, 2023
176a78d
use subshell
matifali Aug 1, 2023
86f22d1
url encoding
matifali Aug 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/pr-cleanup.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,20 @@ jobs:
- name: "Remove PR namespace"
run: |
kubectl delete namespace "pr${{ steps.pr_number.outputs.PR_NUMBER }}" || echo "namespace not found"

- name: "Remove DNS records"
run: |
set -euxo pipefail
# Get identifier for the record
record_id=$(curl -X GET "https://api.cloudflare.com/client/v4/zones/${{ secrets.PR_DEPLOYMENTS_ZONE_ID }}/dns_records?name=*.pr${{ steps.pr_number.outputs.PR_NUMBER }}.${{ secrets.PR_DEPLOYMENTS_DOMAIN }}" \
-H "Authorization: Bearer ${{ secrets.PR_DEPLOYMENTS_CLOUDFLARE_API_TOKEN }}" \
-H "Content-Type:application/json" | jq -r '.result[0].id') || echo "DNS record not found"

echo "::add-mask::$record_id"

# Delete the record
(
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${{ secrets.PR_DEPLOYMENTS_ZONE_ID }}/dns_records/$record_id" \
-H "Authorization: Bearer ${{ secrets.PR_DEPLOYMENTS_CLOUDFLARE_API_TOKEN }}" \
-H "Content-Type:application/json"
) || echo "DNS record not found"
124 changes: 79 additions & 45 deletions .github/workflows/pr-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,18 @@ on:
inputs:
pr_number:
description: "PR number"
type: number
required: true
skip_build:
description: "Skip build job"
required: false
type: boolean
default: false
experiments:
description: "Experiments to enable"
required: false
type: string
default: "*"

env:
REPO: ghcr.io/coder/coder-preview
Expand All @@ -22,8 +29,8 @@ permissions:
pull-requests: write

concurrency:
group: ${{ github.workflow }}-${{ github.event.issue.number || github.run_id }}
cancel-in-progress: false
group: ${{ github.workflow }}-${{ github.repository }}-${{ github.ref }}
cancel-in-progress: true

jobs:
pr_commented:
Expand Down Expand Up @@ -136,7 +143,7 @@ jobs:
PR_TITLE: ${{ needs.pr_commented.outputs.PR_TITLE }}
PR_URL: ${{ needs.pr_commented.outputs.PR_URL }}
PR_BRANCH: ${{ needs.pr_commented.outputs.PR_BRANCH }}
PR_DEPLOYMENT_ACCESS_URL: "https://pr${{ needs.pr_commented.outputs.PR_NUMBER }}.${{ secrets.PR_DEPLOYMENTS_DOMAIN }}"
PR_DEPLOYMENT_ACCESS_URL: "pr${{ needs.pr_commented.outputs.PR_NUMBER }}.${{ secrets.PR_DEPLOYMENTS_DOMAIN }}"
steps:
- name: Check if image exists
run: |
Expand All @@ -145,10 +152,19 @@ jobs:
if [ -z "$foundTag" ]; then
echo "Image not found"
echo "${{ env.CODER_IMAGE_TAG }} not found in ghcr.io/coder/coder-preview"
echo "Please remove --skip-build from the comment or ./scripts/deploy-pr.sh"
echo "Please remove --skip-build from the comment and try again"
exit 1
fi

- name: Add DNS record to Cloudflare
run: |
(
curl -X POST "https://api.cloudflare.com/client/v4/zones/${{ secrets.PR_DEPLOYMENTS_ZONE_ID }}/dns_records" \
-H "Authorization: Bearer ${{ secrets.PR_DEPLOYMENTS_CLOUDFLARE_API_TOKEN }}" \
-H "Content-Type:application/json" \
--data '{"type":"CNAME","name":"*.${{ env.PR_DEPLOYMENT_ACCESS_URL }}","content":"${{ env.PR_DEPLOYMENT_ACCESS_URL }}","ttl":1,"proxied":false}'
)

- name: Checkout
uses: actions/checkout@v3
with:
Expand All @@ -168,35 +184,32 @@ jobs:
kubectl delete namespace "pr${{ env.PR_NUMBER }}" || true
kubectl create namespace "pr${{ env.PR_NUMBER }}"

- name: Setup ingress
- name: Check and Create Certificate
run: |
cat <<EOF > ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: pr${{ env.PR_NUMBER }}
namespace: pr${{ env.PR_NUMBER }}
annotations:
cert-manager.io/cluster-issuer: letsencrypt
spec:
tls:
- hosts:
- "${{ secrets.PR_DEPLOYMENTS_DOMAIN }}"
- "*.${{ secrets.PR_DEPLOYMENTS_DOMAIN }}"
# Using kubectl to check if a Certificate resource already exists
# we are doing this to avoid letsenrypt rate limits
if ! kubectl get certificate pr${{ env.PR_NUMBER }}-tls -n pr-deployment-certs > /dev/null 2>&1; then
echo "Certificate doesn't exist. Creating a new one."
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: pr${{ env.PR_NUMBER }}-tls
namespace: pr-deployment-certs
spec:
secretName: pr${{ env.PR_NUMBER }}-tls
rules:
- host: "pr${{ env.PR_NUMBER }}.${{ secrets.PR_DEPLOYMENTS_DOMAIN }}"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: coder
port:
number: 80
issuerRef:
name: letsencrypt
kind: ClusterIssuer
dnsNames:
- "${{ env.PR_DEPLOYMENT_ACCESS_URL }}"
- "*.${{ env.PR_DEPLOYMENT_ACCESS_URL }}"
EOF
kubectl apply -f ingress.yaml
else
echo "Certificate exists. Skipping certificate creation."
echo "Copy certificate from pr-deployment-certs to pr${{ env.PR_NUMBER }} namespace"
kubectl get secret pr${{ env.PR_NUMBER }}-tls -n pr-deployment-certs -o yaml | sed 's/pr-deployment-certs/pr${{ env.PR_NUMBER }}/g' | kubectl apply -f -
fi

- name: Set up PostgreSQL database
run: |
Expand All @@ -210,6 +223,17 @@ jobs:
kubectl create secret generic coder-db-url -n pr${{ env.PR_NUMBER }} \
--from-literal=url="postgres://coder:coder@coder-db-postgresql.pr${{ env.PR_NUMBER }}.svc.cluster.local:5432/coder?sslmode=disable"

- name: Get experiments
id: get_experiments
run: |
set -euxo pipefail
if [[ ${{ github.event_name }} == "workflow_dispatch" ]]; then
experiments=${{ github.event.inputs.experiments }}
else
experiments=$(echo "${{ github.event.comment.body }}" | grep -oP '(?<=--experiments )[^ ]+')
fi
echo "experiments=$experiments" >> $GITHUB_OUTPUT

- name: Create values.yaml
run: |
cat <<EOF > pr-deploy-values.yaml
Expand All @@ -220,13 +244,22 @@ jobs:
pullPolicy: Always
service:
type: ClusterIP
ingress:
enable: true
className: traefik
host: ${{ env.PR_DEPLOYMENT_ACCESS_URL }}
wildcardHost: "*.${{ env.PR_DEPLOYMENT_ACCESS_URL }}"
tls:
enable: true
secretName: pr${{ env.PR_NUMBER }}-tls
wildcardSecretName: pr${{ env.PR_NUMBER }}-tls
env:
- name: "CODER_ACCESS_URL"
value: "https://pr${{ env.PR_NUMBER }}.${{ secrets.PR_DEPLOYMENTS_DOMAIN }}"
value: "https://${{ env.PR_DEPLOYMENT_ACCESS_URL }}"
- name: "CODER_WILDCARD_ACCESS_URL"
value: "*--pr${{ env.PR_NUMBER }}.${{ secrets.PR_DEPLOYMENTS_DOMAIN }}"
value: "*.${{ env.PR_DEPLOYMENT_ACCESS_URL }}"
- name: "CODER_EXPERIMENTS"
value: "*"
value: "*,${{ steps.get_experiments.outputs.experiments }}"
- name: CODER_PG_CONNECTION_URL
valueFrom:
secretKeyRef:
Expand Down Expand Up @@ -261,7 +294,7 @@ jobs:
set -euxo pipefail

DEST="${HOME}/coder"
URL="${{ env.PR_DEPLOYMENT_ACCESS_URL }}/bin/coder-linux-amd64"
URL="https://${{ env.PR_DEPLOYMENT_ACCESS_URL }}/bin/coder-linux-amd64"

mkdir -p "$(dirname ${DEST})"

Expand All @@ -279,6 +312,7 @@ jobs:
curl -fsSL "$URL" -o "${DEST}"
chmod +x "${DEST}"
"${DEST}" version
mv "${DEST}" /usr/local/bin/coder

- name: Create first user, template and workspace
id: setup_deployment
Expand All @@ -294,16 +328,16 @@ jobs:
echo "::add-mask::$password"
echo "password=$password" >> $GITHUB_OUTPUT

/home/runner/coder login \
--first-user-username pr${{ env.PR_NUMBER }} \
--first-user-email ${{ env.PR_NUMBER }}@coder.com \
coder login \
--first-user-username test \
--first-user-email pr${{ env.PR_NUMBER }}@coder.com \
--first-user-password $password \
--first-user-trial \
--use-token-as-session \
${{ env.PR_DEPLOYMENT_ACCESS_URL }}
https://${{ env.PR_DEPLOYMENT_ACCESS_URL }}

# Create template
/home/runner/coder templates init --id kubernetes && cd ./kubernetes/ && /home/runner/coder templates create -y --variable namespace=pr${{ env.PR_NUMBER }}
coder templates init --id kubernetes && cd ./kubernetes/ && coder templates create -y --variable namespace=pr${{ env.PR_NUMBER }}

# Create workspace
cat <<EOF > workspace.yaml
Expand All @@ -312,8 +346,8 @@ jobs:
home_disk_size: "2"
EOF

/home/runner/coder create --template="kubernetes" pr${{ env.PR_NUMBER }} --rich-parameter-file ./workspace.yaml -y
/home/runner/coder stop pr${{ env.PR_NUMBER }} -y
coder create --template="kubernetes" test --rich-parameter-file ./workspace.yaml -y
coder stop test -y

- name: Send Slack notification
run: |
Expand All @@ -323,9 +357,9 @@ jobs:
"pr_number": "'"${{ env.PR_NUMBER }}"'",
"pr_url": "'"${{ env.PR_URL }}"'",
"pr_title": "'"${{ env.PR_TITLE }}"'",
"pr_access_url": "'"${{ env.PR_DEPLOYMENT_ACCESS_URL }}"'",
"pr_username": "'"pr${{ env.PR_NUMBER }}"'",
"pr_email": "'"${{ env.PR_NUMBER }}@coder.com"'",
"pr_access_url": "'"https://${{ env.PR_DEPLOYMENT_ACCESS_URL }}"'",
"pr_username": "'"test"'",
"pr_email": "'"pr${{ env.PR_NUMBER }}@coder.com"'",
"pr_password": "'"${{ steps.setup_deployment.outputs.password }}"'",
"pr_actor": "'"${{ github.actor }}"'"
}' \
Expand All @@ -351,6 +385,6 @@ jobs:
comment-id: ${{ steps.fc.outputs.comment-id }}
body: |
:heavy_check_mark: Deployed PR ${{ env.PR_NUMBER }} successfully.
:rocket: Access the deployment link [here](${{ env.PR_DEPLOYMENT_ACCESS_URL }}).
:rocket: Access the deployment link [here](https://${{ env.PR_DEPLOYMENT_ACCESS_URL }}).
:warning: This deployment will be deleted when the PR is closed.
reactions: rocket
39 changes: 32 additions & 7 deletions scripts/deploy-pr.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,43 @@ set -euo pipefail
skipBuild=false
dryRun=false
confirm=true
experiments=""

# parse arguments
for arg in "$@"; do
case $arg in
while (("$#")); do
case "$1" in
-s | --skip-build)
skipBuild=true
shift # Remove --skip-build from processing
shift
;;
-n | --dry-run)
dryRun=true
shift # Remove --dry-run from processing
shift
;;
-e | --experiments)
if [ -n "$2" ] && [ "${2:0:1}" != "-" ]; then
experiments="$2"
shift
else
echo "Error: Argument for $1 is missing" >&2
exit 1
fi
shift
;;
-y | --yes)
confirm=false
shift # Remove --yes from processing
shift
;;
--)
shift
break
;;
--*)
echo "Error: Unsupported flag $1" >&2
exit 1
;;
*)
shift # Remove generic argument from processing
shift
;;
esac
done
Expand Down Expand Up @@ -61,7 +80,13 @@ if $dryRun; then
echo "branchName: ${branchName}"
echo "prNumber: ${prNumber}"
echo "skipBuild: ${skipBuild}"
echo "experiments: ${experiments}"
exit 0
fi

gh workflow run pr-deploy.yaml --ref "${branchName}" -f "pr_number=${prNumber}" -f "skip_build=${skipBuild}"
echo "branchName: ${branchName}"
echo "prNumber: ${prNumber}"
echo "skipBuild: ${skipBuild}"
echo "experiments: ${experiments}"

gh workflow run pr-deploy.yaml --ref "${branchName}" -f "pr_number=${prNumber}" -f "skip_build=${skipBuild}" -f "experiments=${experiments}"