-
Notifications
You must be signed in to change notification settings - Fork 254
Description
I'm submitting a ...
- bug report
- feature request
- support request => Please do not submit support request here, see note at the top of this template.
What is the current behavior?
I'd like to show exceptions in Allure report for skipped scenarios
Allure reports show a message and trace when there is a statusDetails
included in the result.json
. It looks like the statusDetails
is only added when the result has an exception. However when you provide an exception for the step, it then returns a status of Failed or Broken even though both may have a status of Skipped
Proposed change
I would like to be able to mark the status as Skipped while still showing the message in the report. A quick peak behind the curtain and it looks like there are a couple ways to accomplish this. The easiest may to be update step_status
to not update result based on type of exception if status is SKIPPED
def step_status(result):
if isinstance(result.status, Enum):
status = STATUS.get(result.status.name, None)
else:
status = STATUS.get(result.status, None)
if status != Status.SKIPPED and result.exception:
status = get_status(result.exception)
return status
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
What is the expected behavior?
I imagine this is the expected behavior. It seems pretty intentional although it's a bummer we cannot associate messages with skipped tests
What is the motivation / use case for changing the behavior?
We have a large test suite and are trying to engage testers / developers to review tests. One pain point is sometimes product defects may take a few weeks or more to get addressed so this adds more noise in the failed / broken results. What we'd like to do is combine custom categories and @known_issue
tags so that we can better categorize skipped scenarios.
Another reason we want to be able to mark these scenarios as skipped (instead of just categorizing types of failures) is we do some post-processing of the results data ourselves for posting back to GitHub / Slack and it would be nice to easily count number of passed, failed, skipped
Here is an example of how this could be used
Please tell us about your environment:
- Allure version: 2.1.0
- Test framework: behave=1.2.6
- Allure adaptor: allure-behave==2.8.24