Description
We already have #2087 about this long request functionality, but that issue has grown so big with all the comments that I decided to create a fresh one. The major difference between the old issue and this one is that while the old one was mainly for discussing about the design, this issue is about actually getting the feature implemented. I now have the high level design ready and explain it below. The plan is to have this feature in RF 4.0 still during 2020.
How to skip tests
There are going to be multiple ways:
- A special exception that library keywords can use to mark a single test to be skipped. See also Expose special exceptions via
robot.api
#3685. - BuiltIn keyword
Skip
(orSkip Test
andSkip Task
) that utilizes the aforementioned exception. - BuiltIn keyowrd
Skip If
to skip based on condition. - When the skipping exception is used in a suite setup, all tests in the suite are skipped.
- Command line option
--skip
to unconditionally skip tests based on tags. Similar to--exclude
but skipped tests are shown in logs/reports with a skip status and not dropped from execution altogether. - Command line option
--skiponfailure
to skip tests if they fail. Similar effect than with the current--noncritical
.
What about criticality
As already discussed in #2087, the skip status is very similar feature than Robot's current criticality concept. There are many people who would like to have both, but I don't think that's a good idea and believe it's better to remove criticality when skipping is added. Separate issue #3624 covers removing criticality and explains this in more detail.
Colors
Skip status needs a specific color to match current pass (green) and fail (red). Yellow feels like a good candidate with a traffic light metaphor, but I'm open for other ideas and we could possibly change other colors as well. Probably should make colors configurable too -- currently only report background colors support it.
Report background color mentioned above needs some thinking as well. Currently it's either green or red, but with the added skip status we could use also yellow or whatever skip color we decide to use. Different scenarios where different colors could be used are listed below (assuming green/yellow/red scheme):
- All tests pass. This is naturally green.
- Any test fails. This is naturally red.
- Any test is skipped (no failures). This probably should be green but could also be yellow.
- All tests skipped. This could be yellow. Could also be green but that's a bit odd if all tests are yellow.