Skip to content

Tests don't fail gracefully when encountering errors after test execution and stall the CI #4211

Closed
@sethvanwykJET

Description

@sethvanwykJET

What are you trying to achieve?

Test pipelines should continue gracefully when encountering an error during "after" test execution

What do you get instead?

When the test fails, codeceptJS fails to properly scale down the test run and keeps the browser session open, essentially turning the test thread into a zombie codeceptJS process.

Tests running in pipelines will time out as the thread that's running these tests don't know the test has completed and that they should start the next test.

   Requesting menu-item sold-out status change to false
      I send put request "https://uri/api/v2/restaurant/1289255/category/15516284/product/487560132", {"name":"Sold Out","description":"Will be changed to sold out by automated tests","external_id":"","sort_id":4,"delivery_price":12,"pickup_price":12,"shipping_method":"pickup_delivery","sold_out":false,"image_url":null,"soft_deleted":false,"addon_set_id":null}, {"accept":"application/json","content-type":"application/json","Authorization":"Basic =="}
      › [Request] {"baseURL":"https://uri/api/v2/restaurant/1289255/category/15516284/product/487560132","method":"PUT","data":{"name":"Sold Out","description":"Will be changed to sold out by automated tests","external_id":"","sort_id":4,"delivery_price":12,"pickup_price":12,"shipping_method":"pickup_delivery","sold_out":false,"image_url":null,"soft_deleted":false,"addon_set_id":null},"headers":{"Content-Type":"application/json","Accept":"application/json","X-Session-Id":"8c7db2c0a86b","user_agent":"cw-api-automated-tests","sec-fetch-mode":"cors","access-control-allow-origin":"*","accept":"application/json","content-type":"application/json","Authorization":"Basic =="}}
      › [Response] Response error. Status code: 404
      › [Response] "404 page not found\n"
404 page not found

    [2] Error | Error: Restaurant Menu API call failed! Response: undefined
             Status: 404 (e) => { const err = (recorder.getAsyncErr() === n...
Error processing test.after event:
TypeError: Cannot read properties of undefined (reading 'ctx')
    at AllureReporter.allureTestByCodeceptTest (/Users/seth.vanwyk/workspace/project/node_modules/allure-codeceptjs/dist/index.js:89:62)
    at AllureReporter.testAfter (/Users/seth.vanwyk/workspace/project/node_modules/allure-codeceptjs/dist/index.js:159:33)
    at EventEmitter.emit (node:events:402:35)
    at Object.emit (/Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/event.js:149:28)
    at errHandler (/Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/scenario.js:116:39)
    at /Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/scenario.js:162:11
    at /Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/recorder.js:239:9
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    [2] <teardown> Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › Screenshot is saving to /Users/seth.vanwyk/workspace/project/output/Desktop_-_item_is_sold_out_after_each_hook__After_for_should_remove_item_that_is_sold_out_from_basket_@site-nl_@delivery_@not-allowed-to-run-in-production_@should-remove-item-that-is-sold-out-from-basket.failed.png
  ✖ FAILED in 434ms

    [3]  Starting recording promises

-- FAILURES:

  1) Desktop - item is sold out
       should remove item that is sold out from basket @site-nl @delivery @not-allowed-to-run-in-production @should-remove-item-that-is-sold-out-from-basket:
     Error: Restaurant Menu API call failed! Response: undefined
             Status: 404
      at Object.changeMenuItemSoldOutStatus (api-calls/restaurant-menu-api.js:36:15)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Test.<anonymous> (/Users/seth.vanwyk/workspace/project/checkout-e2e/scenarios/multi-step-checkout/item-out-is-sold-out-alert-desktop-test.js:52:3)

  Scenario Steps:
  - I.seeElement({"xpath":"//*[@data-qa='jet-privacy-settings-button-full-consent']"}) at ./page-objects/footer/footer.js:16:41
  - I.seeElement({"xpath":"//*[@data-qa='privacy-settings-action-close']"}) at ./page-objects/footer/footer.js:15:42
  - I.say("Checking which cookie banner variant is in use") at Object.acceptCookies (./page-objects/footer/footer.js:14:7)
  - I.say("Opened page https://site/en/menu/test-restaurant-selenium?showTestRestaurants=true&utm_medium=e2e_testing&utm_source=cwplatform&utm_campaign=local&e2e=true&timestamp=1708522969025 at Wed Feb 21 2024 13:42:49 GMT+0000 (Coordinated Universal Time)") at Actor.accessURL (./steps-file.js:174:12)
  - I.grabCurrentUrl() at Actor.accessURL (./steps-file.js:173:37)
  - I.waitForInvisible({xpath: //*[contains(@data-qa,'skeleton')]}, 30) at ./steps-file.js:210:18
  - I.say("Awaiting for 30 seconds for skeleton to not be visible") at Actor.waitPageLoads (./steps-file.js:206:14)
  - I.waitForFunction(async () => {
          const state = await document.readyState;
          return state === 'interactive' || state === 'complete';
        }, 120) at Actor.waitPageLoads (./steps-file.js:197:12)
  - I.say("Awaiting for page to load with javascript") at Actor.waitPageLoads (./steps-file.js:196:12)
  - I.amOnPage("https://site/en/menu/test-restaurant-selenium?lat=53.0651908&lng=16.04760159&showTestRestaurants=true&utm_medium=e2e_testing&utm_source=cwplatform&utm_campaign=local&e2e=true&timestamp=1708522969025") at Actor.accessURL (./steps-file.js:167:14)
  - I.say("Opening the page") at Actor.accessURL (./steps-file.js:164:12)
  - I.sendGetRequest("/restaurant?slug=test-restaurant-selenium", {"X-Country-Code":"pl","Accept-Language":"en","X-Language-Code":"en"}) at getTakeawayRestaurantOpening (./api-calls/restaurant.js:6:28)

  Artifacts:
  - screenshot: /Users/seth.vanwyk/workspace/project/output/should_remove_item_that_is_sold_out_from_basket_@site-nl_@delivery_@not-allowed-to-run-in-production_@should-remove-item-that-is-sold-out-from-basket.failed.png

  2) Desktop - item is sold out
       "after each" hook: After for "should remove item that is sold out from basket @site-nl @delivery @not-allowed-to-run-in-production @should-remove-item-that-is-sold-out-from-basket":
     Error: Restaurant Menu API call failed! Response: undefined
             Status: 404
      at Object.changeMenuItemSoldOutStatus (api-calls/restaurant-menu-api.js:36:15)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (/Users/seth.vanwyk/workspace/project/checkout-e2e/scenarios/multi-step-checkout/item-out-is-sold-out-alert-desktop-test.js:75:3)
      at async promiseRetry.retries.retries (/Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/scenario.js:143:9)




  FAIL  | 0 passed, 2 failed   // 27s

The terminal stalls here and leaves a broken browser session open that doesn't close anymore.

Provide test source code if related

    if (response.data.message !== 'success') {
        console.log(response.data)
        throw new Error(
            `Restaurant Menu API call failed! Response: ${response.data.message}
             Status: ${response.status}`
        );

Details

  • CodeceptJS version: Tested on 3.5.3 and 3.5.14
  • NodeJS Version:16
  • Operating System: MacOS
  • puppeteer

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions