Skip to content

chore: enhance tests for TemplateSchedulePage #9801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Sep 21, 2023
Merged

chore: enhance tests for TemplateSchedulePage #9801

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cfc53fb
chore: Add benchmark logs to test
Parkreiner Sep 20, 2023
3191ddb
chore: Remove benchmark logic
Parkreiner Sep 20, 2023
56873b8
chore: add hard cutoff for waitFor calls
Parkreiner Sep 20, 2023
959f5fc
refactor: clean up waitFor cut-off logic
Parkreiner Sep 20, 2023
62fe36c
chore: add assertion that submit button is not disabled
Parkreiner Sep 20, 2023
1cfedaf
Merge branch 'main' into template-schedule-flake
Parkreiner Sep 20, 2023
507dc80
chore: Remove disabled check at the start of the test
Parkreiner Sep 20, 2023
c4fff90
fix: extend cutoff for waitFor config
Parkreiner Sep 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 49 additions & 21 deletions site/src/pages/TemplateSettingsPage/TemplateSchedulePage/TemplateSchedulePage.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,19 +36,22 @@ const renderTemplateSchedulePage = async () => {
await waitForLoaderToBeRemoved();
};

// Extracts all properties from TemplateScheduleFormValues that have a key that
// ends in _ms, and makes those properties optional. Defined as mapped type to
// ensure this stays in sync as TemplateScheduleFormValues changes
type FillAndSubmitConfig = {
[Key in keyof TemplateScheduleFormValues as Key extends `${string}_ms`
? Key
: never]?: TemplateScheduleFormValues[Key] | undefined;
};

const fillAndSubmitForm = async ({
default_ttl_ms,
max_ttl_ms,
failure_ttl_ms,
time_til_dormant_ms,
time_til_dormant_autodelete_ms,
}: {
default_ttl_ms?: number;
max_ttl_ms?: number;
failure_ttl_ms?: number;
time_til_dormant_ms?: number;
time_til_dormant_autodelete_ms?: number;
}) => {
}: FillAndSubmitConfig) => {
const user = userEvent.setup();

if (default_ttl_ms) {
Expand All @@ -61,6 +64,7 @@ const fillAndSubmitForm = async ({

if (max_ttl_ms) {
const maxTtlField = await screen.findByLabelText("Max lifetime (hours)");

await user.clear(maxTtlField);
await user.type(maxTtlField, max_ttl_ms.toString());
}
Expand Down Expand Up @@ -89,16 +93,31 @@ const fillAndSubmitForm = async ({
);
}

const submitButton = await screen.findByText(
FooterFormLanguage.defaultSubmitLabel,
);
const submitButton = screen.getByRole("button", {
name: FooterFormLanguage.defaultSubmitLabel,
});

expect(submitButton).not.toBeDisabled();
await user.click(submitButton);

// User needs to confirm dormancy and autodeletion fields.
// User needs to confirm dormancy and auto-deletion fields.
const confirmButton = await screen.findByTestId("confirm-button");
await user.click(confirmButton);
};

// One problem with the waitFor function is that if no additional config options
// are passed in, it will hang indefinitely as it keeps retrying an assertion.
// Even if Jest runs out of time and kills the test, you won't get a good error
// message. Adding options to force test to give up before test timeout
function waitForWithCutoff(callback: () => void | Promise<void>) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I get the idea behind waitForWithCutoff. In your setup it could be 13sec, but for somebody else, it might be 17sec. I mean, we shouldn't create more time-dependent tests unless there is no way to prevent an event.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's fair. I keep forgetting that I'm on a Macbook, so the performance is going to be a lot faster. I also agree with you – ideally we shouldn't be introducing time-dependent tests unnecessarily

My worry is that with my understanding of the base waitFor, it doesn't fail without some kind of configuration object. As in, it will try to make its assertion periodically, but if the assertion fails, it will just go back to waiting – that failed assertion will just get caught and discarded. So if the assertion doesn't succeed before Jest's global cutoff time runs out, we just get the generic timeout message, instead of something more specific. With some kind of timeout in place, waitFor will eventually stop swallowing the failed assertion and throw it to the test suite

I have two other ideas, though

  1. Export the global timeout value as a constant, and define the tests to have a hard cutoff of half a second before time is up
  2. Just try extending timeouts for the specific tests by passing a custom timeout value to the describe functions

I'd be leaning more towards number 1, but they both feel varying degrees of hack-y still

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm just biased as we have dealt a lot with increasing timeouts endlessly on the backend side to make tests pass. Usually, the clue was to refactor or split the test into 2 separate paths.

I will leave the decision to you folks! cc @BrunoQuaresma

Copy link
Collaborator

@BrunoQuaresma BrunoQuaresma Sep 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, there is no good answer for this 😞 since the test environment for FE tests can be "not very reliable". Maybe we could spend some time trying to figure out better test alternatives but IMO it would be a diff task.

return waitFor(callback, {
// Defined to end 500ms before global cut-off time of 20s. Wanted to define
// this in terms of an exported constant from jest.config, but since Jest
// is CJS-based, that would've involved weird CJS-ESM interop issues
timeout: 19_500,
});
}

describe("TemplateSchedulePage", () => {
beforeEach(() => {
jest
Expand All @@ -109,14 +128,17 @@ describe("TemplateSchedulePage", () => {
jest.spyOn(API, "getExperiments").mockResolvedValue(["workspace_actions"]);
});

it("succeeds", async () => {
it("Calls the API when user fills in and submits a form", async () => {
await renderTemplateSchedulePage();
jest.spyOn(API, "updateTemplateMeta").mockResolvedValueOnce({
...MockTemplate,
...validFormValues,
});

await fillAndSubmitForm(validFormValues);
await waitFor(() => expect(API.updateTemplateMeta).toBeCalledTimes(1));
await waitForWithCutoff(() =>
expect(API.updateTemplateMeta).toBeCalledTimes(1),
);
});

test("default and max ttl is converted to and from hours", async () => {
Expand All @@ -128,16 +150,19 @@ describe("TemplateSchedulePage", () => {
});

await fillAndSubmitForm(validFormValues);
await waitFor(() => expect(API.updateTemplateMeta).toBeCalledTimes(1));
await waitFor(() =>
await waitForWithCutoff(() =>
expect(API.updateTemplateMeta).toBeCalledTimes(1),
);

await waitForWithCutoff(() => {
expect(API.updateTemplateMeta).toBeCalledWith(
"test-template",
expect.objectContaining({
default_ttl_ms: (validFormValues.default_ttl_ms || 0) * 3600000,
max_ttl_ms: (validFormValues.max_ttl_ms || 0) * 3600000,
}),
),
);
);
});
});

test("failure, dormancy, and dormancy auto-deletion converted to and from days", async () => {
Expand All @@ -149,8 +174,11 @@ describe("TemplateSchedulePage", () => {
});

await fillAndSubmitForm(validFormValues);
await waitFor(() => expect(API.updateTemplateMeta).toBeCalledTimes(1));
await waitFor(() =>
await waitForWithCutoff(() =>
expect(API.updateTemplateMeta).toBeCalledTimes(1),
);

await waitForWithCutoff(() => {
expect(API.updateTemplateMeta).toBeCalledWith(
"test-template",
expect.objectContaining({
Expand All @@ -160,8 +188,8 @@ describe("TemplateSchedulePage", () => {
time_til_dormant_autodelete_ms:
(validFormValues.time_til_dormant_autodelete_ms || 0) * 86400000,
}),
),
);
);
});
});

it("allows a default ttl of 7 days", () => {
Expand Down