Skip to content

Support custom statuses #3625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pekkaklarck opened this issue Jul 31, 2020 · 8 comments
Open

Support custom statuses #3625

pekkaklarck opened this issue Jul 31, 2020 · 8 comments

Comments

@pekkaklarck
Copy link
Member

It would be convenient to be able to configure Robot Framework to have custom statuses in addition to the current PASS and FAIL and the forthcoming SKIP (#3622). Currently tests can have tags to differentiate them, but explicit statuses would make the distinction more clear.

This functionality could be implemented so that new statuses are defined based on old statuses and tags. The specification could be given from the command like with a command line option like --addstatus with an argument in format such as NEWSTATUS:OLDSTATUS:TAGPATTERN:COLOR. In this spec NEWSTATUS would be the name of the new status to add, OLDSTATUS would be the old status to base this status on, TAGPATTERN would be the tag or tag pattern (e.g. tag, tag:* or tag1ORtag2) matching test must have, and COLOR would be the color associated with the status (possibly optional). Example:

--addstatus KNOWN_ISSUE:FAIL:bug-id-*:purple

One use case for this feature would be separating tests skipped explicitly during execution and tests skipped by the forthcoming --skiponfailure feature. This would be convenient for those who oppose removing the criticality concept (#3624):

--addstatus NON-CRITICAL:SKIP:non-critical:pink --skiponfailure non-critical
@pekkaklarck
Copy link
Member Author

This would be nice already in RF 4.1.

@pekkaklarck pekkaklarck added this to the v4.1 milestone Mar 12, 2021
@pekkaklarck pekkaklarck modified the milestones: v4.1, v5.0 Jun 7, 2021
@pekkaklarck
Copy link
Member Author

Too big task for RF 4.1. I moved this tentatively to RF 5.0 scope but we haven't yet started more detailed planning of that release and it's not clear will this one be actually included or not. If you find this important, comment here and explain your use case!

@pekkaklarck pekkaklarck modified the milestones: v5.0, v4.1 Jun 7, 2021
@pekkaklarck
Copy link
Member Author

Oooops, there's PR #3869 adding this functionality. If the PR implements what's needed and doesn't have problems, this certainly can be included into RF 4.1 I don't have time to review the PR right now, but I moved this issue back to RF 4.1 to make sure I don't forget about it.

@pekkaklarck
Copy link
Member Author

PR #3869 is a good start. There is, however, so much work still to be done both internally to handle custom statuses and in log report to show them nicely in statistics and elsewhere that I move this issue to RF 5.0 scope.

@pekkaklarck pekkaklarck modified the milestones: v4.1, v5.0 Jun 23, 2021
@Tattoo
Copy link
Member

Tattoo commented Nov 9, 2021

I would like to add here a wish that custom statuses would be recorded in the output.xml in some way (no opinion on how exactly) that for example the RF Jenkins plugin could easily still show only PASS/FAIL/SKIP, even if the report itself has custom statuses. We try to keep the statistics minimal (for backwards support), and CICD anyway (outside of RF Jenkins plugin) will rely on the return code of RF to know if execution ought to be stopped or not.

Given RF Jenkins Plugin is in Java, meaning we cannot leverage the parser, it means we have to support this by parsing the output.xml ourselves.

@pekkaklarck
Copy link
Member Author

I just rechecked PR #3869 and noticed that there's even more work, including big design decisions, to be made for this to be reasonable enhancement in RF 5.0. In fact, it's way too big even for RF 5.1 that is going to b a relatively small release. On top of that, there hasn't been too much interest towards this issue. I still consider this a nice enhancement but we can return to it later if there's more interest.

@pekkaklarck pekkaklarck removed this from the v5.0 milestone Mar 10, 2022
@Moses501
Copy link

Hi Pekka Klärck !
it's a very interesting feature !
do we know when it could be developed?

@pekkaklarck
Copy link
Member Author

This feature isn't currently targeted for any planned release. If someone is interested to look at developing it, I'd be happy to help. Paying for the development is possible as well. The best way to get started with that would be joining the Robot Framework Foundation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants