Skip to content

Show exceptions in Allure report for skipped scenarios #580

@JoelWhitney

Description

@JoelWhitney

I'm submitting a ...

  • bug report
  • feature request
  • support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

I'd like to show exceptions in Allure report for skipped scenarios

Allure reports show a message and trace when there is a statusDetails included in the result.json. It looks like the statusDetails is only added when the result has an exception. However when you provide an exception for the step, it then returns a status of Failed or Broken even though both may have a status of Skipped

Proposed change

I would like to be able to mark the status as Skipped while still showing the message in the report. A quick peak behind the curtain and it looks like there are a couple ways to accomplish this. The easiest may to be update step_status to not update result based on type of exception if status is SKIPPED

def step_status(result):
    if isinstance(result.status, Enum):
        status = STATUS.get(result.status.name, None)
    else:
        status = STATUS.get(result.status, None)
    if status != Status.SKIPPED and result.exception:
        status = get_status(result.exception)
    return status

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

What is the expected behavior?

I imagine this is the expected behavior. It seems pretty intentional although it's a bummer we cannot associate messages with skipped tests

What is the motivation / use case for changing the behavior?

We have a large test suite and are trying to engage testers / developers to review tests. One pain point is sometimes product defects may take a few weeks or more to get addressed so this adds more noise in the failed / broken results. What we'd like to do is combine custom categories and @known_issue tags so that we can better categorize skipped scenarios.

Another reason we want to be able to mark these scenarios as skipped (instead of just categorizing types of failures) is we do some post-processing of the results data ourselves for posting back to GitHub / Slack and it would be nice to easily count number of passed, failed, skipped

Here is an example of how this could be used

image

Please tell us about your environment:

  • Allure version: 2.1.0
  • Test framework: behave=1.2.6
  • Allure adaptor: allure-behave==2.8.24

Other information

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions