Skip to content

chore: add workspace reached resource threshold notification #16250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Feb 4, 2025
Merged

chore: add workspace reached resource threshold notification #16250

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6c6240b
chore: add workspace reached resource threshold notification
DanielleMaywood Jan 24, 2025
b3081de
chore: split out into two notifications
DanielleMaywood Jan 30, 2025
a9c8676
chore: update golden files
DanielleMaywood Jan 30, 2025
4c21ce7
Merge branch 'main' into dm-internal-247
DanielleMaywood Feb 4, 2025
944fdb5
chore: rewrite ood notification
DanielleMaywood Feb 4, 2025
6444176
chore: updaten golden file
DanielleMaywood Feb 4, 2025
bc87268
chore: silly me
DanielleMaywood Feb 4, 2025
d2265f6
chore: rename test
DanielleMaywood Feb 4, 2025
4d38b1f
Merge branch 'main' into dm-internal-247
DanielleMaywood Feb 4, 2025
9fd5d2b
chore: bump migration number, delete old files
DanielleMaywood Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions coderd/database/migrations/000288_oom_and_ood_notification.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
DELETE FROM notification_templates WHERE id = 'f047f6a3-5713-40f7-85aa-0394cce9fa3a';
DELETE FROM notification_templates WHERE id = 'a9d027b4-ac49-4fb1-9f6d-45af15f64e7a';
33 changes: 33 additions & 0 deletions coderd/database/migrations/000288_oom_and_ood_notification.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
INSERT INTO notification_templates
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make these inactive by default? We'll be launching inbox (hopefully) soon after OOM/OOD drops, and duplication would be unfortunate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do, although I could be convinced to go either way on this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stirby WDYT? I fairly feel strongly that SMTP alerts for these are low value, and webhook is unlikely to be the default method chosen most of the time.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to leave these enabled for user visibility. I think more redundancy is preferred since these are pointing out an error that may lead to loss of work.

(id, name, title_template, body_template, "group", actions)
VALUES (
'a9d027b4-ac49-4fb1-9f6d-45af15f64e7a',
'Workspace Out Of Memory',
E'Your workspace "{{.Labels.workspace}}" is low on memory',
E'Hi {{.UserName}},\n\n'||
E'Your workspace **{{.Labels.workspace}}** has reached the memory usage threshold set at **{{.Labels.threshold}}**.',
'Workspace Events',
'[
{
"label": "View workspace",
"url": "{{base_url}}/@{{.UserUsername}}/{{.Labels.workspace}}"
}
]'::jsonb
);

INSERT INTO notification_templates
(id, name, title_template, body_template, "group", actions)
VALUES (
'f047f6a3-5713-40f7-85aa-0394cce9fa3a',
'Workspace Out Of Disk',
E'Your workspace "{{.Labels.workspace}}" is low on disk',
E'Hi {{.UserName}},\n\n'||
E'Your workspace **{{.Labels.workspace}}** has reached the usage threshold set at **{{.Labels.threshold}}** for volume `{{.Labels.volume}}`.',
'Workspace Events',
'[
{
"label": "View workspace",
"url": "{{base_url}}/@{{.UserUsername}}/{{.Labels.workspace}}"
}
]'::jsonb
);
2 changes: 2 additions & 0 deletions coderd/notifications/events.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ var (
TemplateWorkspaceAutoUpdated = uuid.MustParse("c34a0c09-0704-4cac-bd1c-0c0146811c2b")
TemplateWorkspaceMarkedForDeletion = uuid.MustParse("51ce2fdf-c9ca-4be1-8d70-628674f9bc42")
TemplateWorkspaceManualBuildFailed = uuid.MustParse("2faeee0f-26cb-4e96-821c-85ccb9f71513")
TemplateWorkspaceOutOfMemory = uuid.MustParse("a9d027b4-ac49-4fb1-9f6d-45af15f64e7a")
TemplateWorkspaceOutOfDisk = uuid.MustParse("f047f6a3-5713-40f7-85aa-0394cce9fa3a")
)

// Account-related events.
Expand Down
27 changes: 27 additions & 0 deletions coderd/notifications/notifications_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1064,6 +1064,33 @@ func TestNotificationTemplates_Golden(t *testing.T) {
},
},
},
{
name: "TemplateWorkspaceOutOfMemory",
id: notifications.TemplateWorkspaceOutOfMemory,
payload: types.MessagePayload{
UserName: "Bobby",
UserEmail: "bobby@coder.com",
UserUsername: "bobby",
Labels: map[string]string{
"workspace": "bobby-workspace",
"threshold": "90%",
},
},
},
{
name: "TemplateWorkspaceOutOfDisk",
id: notifications.TemplateWorkspaceOutOfDisk,
payload: types.MessagePayload{
UserName: "Bobby",
UserEmail: "bobby@coder.com",
UserUsername: "bobby",
Labels: map[string]string{
"workspace": "bobby-workspace",
"threshold": "90%",
"volume": "/home/coder",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we only alert on one volume being above threshold at once?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With how this template is written, yes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we refactor to accept a slice of volumes please?
As an end-user, I wouldn't find much value in receiving one notification for each volume. We already have memory and volume split which I understand is for practical reasons, but splitting volumes would just add noise.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have now done so, have updated the screenshots to show the update notification wording

},
},
},
}

// We must have a test case for every notification_template. This is enforced below:
Expand Down
80 changes: 80 additions & 0 deletions coderd/notifications/testdata/rendered-templates/smtp/TemplateWorkspaceOutOfDisk.html.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
From: system@coder.com
To: bobby@coder.com
Subject: Your workspace "bobby-workspace" is low on disk
Message-Id: 02ee4935-73be-4fa1-a290-ff9999026b13@blush-whale-48
Date: Fri, 11 Oct 2024 09:03:06 +0000
Content-Type: multipart/alternative; boundary=bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
MIME-Version: 1.0

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

Hi Bobby,

Your workspace bobby-workspace has reached the usage threshold set at 90% f=
or volume /home/coder.


View workspace: http://test.com/@bobby/bobby-workspace

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8

<!doctype html>
<html lang=3D"en">
<head>
<meta charset=3D"UTF-8" />
<meta name=3D"viewport" content=3D"width=3Ddevice-width, initial-scale=
=3D1.0" />
<title>Your workspace "bobby-workspace" is low on disk</title>
</head>
<body style=3D"margin: 0; padding: 0; font-family: -apple-system, system-=
ui, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarel=
l', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; color: #020617=
; background: #f8fafc;">
<div style=3D"max-width: 600px; margin: 20px auto; padding: 60px; borde=
r: 1px solid #e2e8f0; border-radius: 8px; background-color: #fff; text-alig=
n: left; font-size: 14px; line-height: 1.5;">
<div style=3D"text-align: center;">
<img src=3D"https://coder.com/coder-logo-horizontal.png" alt=3D"Cod=
er Logo" style=3D"height: 40px;" />
</div>
<h1 style=3D"text-align: center; font-size: 24px; font-weight: 400; m=
argin: 8px 0 32px; line-height: 1.5;">
Your workspace "bobby-workspace" is low on disk
</h1>
<div style=3D"line-height: 1.5;">
<p>Hi Bobby,</p>

<p>Your workspace <strong>bobby-workspace</strong> has reached the usage th=
reshold set at <strong>90%</strong> for volume <code>/home/coder</code>.</p=
>
</div>
<div style=3D"text-align: center; margin-top: 32px;">
=20
<a href=3D"http://test.com/@bobby/bobby-workspace" style=3D"display=
: inline-block; padding: 13px 24px; background-color: #020617; color: #f8fa=
fc; text-decoration: none; border-radius: 8px; margin: 0 4px;">
View workspace
</a>
=20
</div>
<div style=3D"border-top: 1px solid #e2e8f0; color: #475569; font-siz=
e: 12px; margin-top: 64px; padding-top: 24px; line-height: 1.6;">
<p>&copy;&nbsp;2024&nbsp;Coder. All rights reserved&nbsp;-&nbsp;<a =
href=3D"http://test.com" style=3D"color: #2563eb; text-decoration: none;">h=
ttp://test.com</a></p>
<p><a href=3D"http://test.com/settings/notifications" style=3D"colo=
r: #2563eb; text-decoration: none;">Click here to manage your notification =
settings</a></p>
<p><a href=3D"http://test.com/settings/notifications?disabled=3Df04=
7f6a3-5713-40f7-85aa-0394cce9fa3a" style=3D"color: #2563eb; text-decoration=
: none;">Stop receiving emails like this</a></p>
</div>
</div>
</body>
</html>

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4--
79 changes: 79 additions & 0 deletions ...d/notifications/testdata/rendered-templates/smtp/TemplateWorkspaceOutOfMemory.html.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
From: system@coder.com
To: bobby@coder.com
Subject: Your workspace "bobby-workspace" is low on memory
Message-Id: 02ee4935-73be-4fa1-a290-ff9999026b13@blush-whale-48
Date: Fri, 11 Oct 2024 09:03:06 +0000
Content-Type: multipart/alternative; boundary=bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
MIME-Version: 1.0

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

Hi Bobby,

Your workspace bobby-workspace has reached the memory usage threshold set a=
t 90%.


View workspace: http://test.com/@bobby/bobby-workspace

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8

<!doctype html>
<html lang=3D"en">
<head>
<meta charset=3D"UTF-8" />
<meta name=3D"viewport" content=3D"width=3Ddevice-width, initial-scale=
=3D1.0" />
<title>Your workspace "bobby-workspace" is low on memory</title>
</head>
<body style=3D"margin: 0; padding: 0; font-family: -apple-system, system-=
ui, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarel=
l', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; color: #020617=
; background: #f8fafc;">
<div style=3D"max-width: 600px; margin: 20px auto; padding: 60px; borde=
r: 1px solid #e2e8f0; border-radius: 8px; background-color: #fff; text-alig=
n: left; font-size: 14px; line-height: 1.5;">
<div style=3D"text-align: center;">
<img src=3D"https://coder.com/coder-logo-horizontal.png" alt=3D"Cod=
er Logo" style=3D"height: 40px;" />
</div>
<h1 style=3D"text-align: center; font-size: 24px; font-weight: 400; m=
argin: 8px 0 32px; line-height: 1.5;">
Your workspace "bobby-workspace" is low on memory
</h1>
<div style=3D"line-height: 1.5;">
<p>Hi Bobby,</p>

<p>Your workspace <strong>bobby-workspace</strong> has reached the memory u=
sage threshold set at <strong>90%</strong>.</p>
</div>
<div style=3D"text-align: center; margin-top: 32px;">
=20
<a href=3D"http://test.com/@bobby/bobby-workspace" style=3D"display=
: inline-block; padding: 13px 24px; background-color: #020617; color: #f8fa=
fc; text-decoration: none; border-radius: 8px; margin: 0 4px;">
View workspace
</a>
=20
</div>
<div style=3D"border-top: 1px solid #e2e8f0; color: #475569; font-siz=
e: 12px; margin-top: 64px; padding-top: 24px; line-height: 1.6;">
<p>&copy;&nbsp;2024&nbsp;Coder. All rights reserved&nbsp;-&nbsp;<a =
href=3D"http://test.com" style=3D"color: #2563eb; text-decoration: none;">h=
ttp://test.com</a></p>
<p><a href=3D"http://test.com/settings/notifications" style=3D"colo=
r: #2563eb; text-decoration: none;">Click here to manage your notification =
settings</a></p>
<p><a href=3D"http://test.com/settings/notifications?disabled=3Da9d=
027b4-ac49-4fb1-9f6d-45af15f64e7a" style=3D"color: #2563eb; text-decoration=
: none;">Stop receiving emails like this</a></p>
</div>
</div>
</body>
</html>

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4--
79 changes: 79 additions & 0 deletions ...ns/testdata/rendered-templates/smtp/TemplateWorkspaceReachedResourceThreshold.html.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
From: system@coder.com
To: bobby@coder.com
Subject: Workspace "bobby-workspace" reached resource threshold
Message-Id: 02ee4935-73be-4fa1-a290-ff9999026b13@blush-whale-48
Date: Fri, 11 Oct 2024 09:03:06 +0000
Content-Type: multipart/alternative; boundary=bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
MIME-Version: 1.0

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

Hi Bobby,

Your workspace bobby-workspace has reached the memory usage threshold set a=
t 90%.


View workspace: http://test.com/@bobby/bobby-workspace

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8

<!doctype html>
<html lang=3D"en">
<head>
<meta charset=3D"UTF-8" />
<meta name=3D"viewport" content=3D"width=3Ddevice-width, initial-scale=
=3D1.0" />
<title>Workspace "bobby-workspace" reached resource threshold</title>
</head>
<body style=3D"margin: 0; padding: 0; font-family: -apple-system, system-=
ui, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarel=
l', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; color: #020617=
; background: #f8fafc;">
<div style=3D"max-width: 600px; margin: 20px auto; padding: 60px; borde=
r: 1px solid #e2e8f0; border-radius: 8px; background-color: #fff; text-alig=
n: left; font-size: 14px; line-height: 1.5;">
<div style=3D"text-align: center;">
<img src=3D"https://coder.com/coder-logo-horizontal.png" alt=3D"Cod=
er Logo" style=3D"height: 40px;" />
</div>
<h1 style=3D"text-align: center; font-size: 24px; font-weight: 400; m=
argin: 8px 0 32px; line-height: 1.5;">
Workspace "bobby-workspace" reached resource threshold
</h1>
<div style=3D"line-height: 1.5;">
<p>Hi Bobby,</p>

<p>Your workspace <strong>bobby-workspace</strong> has reached the memory u=
sage threshold set at <strong>90%</strong>.</p>
</div>
<div style=3D"text-align: center; margin-top: 32px;">
=20
<a href=3D"http://test.com/@bobby/bobby-workspace" style=3D"display=
: inline-block; padding: 13px 24px; background-color: #020617; color: #f8fa=
fc; text-decoration: none; border-radius: 8px; margin: 0 4px;">
View workspace
</a>
=20
</div>
<div style=3D"border-top: 1px solid #e2e8f0; color: #475569; font-siz=
e: 12px; margin-top: 64px; padding-top: 24px; line-height: 1.6;">
<p>&copy;&nbsp;2024&nbsp;Coder. All rights reserved&nbsp;-&nbsp;<a =
href=3D"http://test.com" style=3D"color: #2563eb; text-decoration: none;">h=
ttp://test.com</a></p>
<p><a href=3D"http://test.com/settings/notifications" style=3D"colo=
r: #2563eb; text-decoration: none;">Click here to manage your notification =
settings</a></p>
<p><a href=3D"http://test.com/settings/notifications?disabled=3Da9d=
027b4-ac49-4fb1-9f6d-45af15f64e7a" style=3D"color: #2563eb; text-decoration=
: none;">Stop receiving emails like this</a></p>
</div>
</div>
</body>
</html>

--bbe61b741255b6098bb6b3c1f41b885773df633cb18d2a3002b68e4bc9c4--
29 changes: 29 additions & 0 deletions .../notifications/testdata/rendered-templates/webhook/TemplateWorkspaceOutOfDisk.json.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"_version": "1.1",
"msg_id": "00000000-0000-0000-0000-000000000000",
"payload": {
"_version": "1.1",
"notification_name": "Workspace Out Of Disk",
"notification_template_id": "00000000-0000-0000-0000-000000000000",
"user_id": "00000000-0000-0000-0000-000000000000",
"user_email": "bobby@coder.com",
"user_name": "Bobby",
"user_username": "bobby",
"actions": [
{
"label": "View workspace",
"url": "http://test.com/@bobby/bobby-workspace"
}
],
"labels": {
"threshold": "90%",
"volume": "/home/coder",
"workspace": "bobby-workspace"
},
"data": null
},
"title": "Your workspace \"bobby-workspace\" is low on disk",
"title_markdown": "Your workspace \"bobby-workspace\" is low on disk",
"body": "Hi Bobby,\n\nYour workspace bobby-workspace has reached the usage threshold set at 90% for volume /home/coder.",
"body_markdown": "Hi Bobby,\n\nYour workspace **bobby-workspace** has reached the usage threshold set at **90%** for volume `/home/coder`."
}
28 changes: 28 additions & 0 deletions ...otifications/testdata/rendered-templates/webhook/TemplateWorkspaceOutOfMemory.json.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"_version": "1.1",
"msg_id": "00000000-0000-0000-0000-000000000000",
"payload": {
"_version": "1.1",
"notification_name": "Workspace Out Of Memory",
"notification_template_id": "00000000-0000-0000-0000-000000000000",
"user_id": "00000000-0000-0000-0000-000000000000",
"user_email": "bobby@coder.com",
"user_name": "Bobby",
"user_username": "bobby",
"actions": [
{
"label": "View workspace",
"url": "http://test.com/@bobby/bobby-workspace"
}
],
"labels": {
"threshold": "90%",
"workspace": "bobby-workspace"
},
"data": null
},
"title": "Your workspace \"bobby-workspace\" is low on memory",
"title_markdown": "Your workspace \"bobby-workspace\" is low on memory",
"body": "Hi Bobby,\n\nYour workspace bobby-workspace has reached the memory usage threshold set at 90%.",
"body_markdown": "Hi Bobby,\n\nYour workspace **bobby-workspace** has reached the memory usage threshold set at **90%**."
}
Loading
Loading