Robot Framework currently has a concept of criticality that makes it possible to run tests so that their failures don't affect the overall test execution verdict. The motivation for this feature is acceptance test driven development (ATDD) where you create tests before implementing features and those tests naturally cannot pass initially. In addition to that, this functionality has been used for emulating skipping tests by dynamically marking them non-critical before failing. The system works by using --critical and --noncritical options matching tests by tags.

Although this functionality has worked ok in its designed usage, it also has several problems and it will also become redundant when RF 4.0 adds real skip status (#3622). This issue first goes through the aforementioned problems in more detail and then explains how the forthcoming skip functionality can be used in the ATDD context to get same functionality that criticality currently provides.

Problems with criticality

1. RF 4.0 will introduce real skip status (New SKIP status #3622) which is conceptually very close to the criticality functionality. There are some differences, but these featues are so close that having both doesn't add much benefits but instead causes confusion and adds unnecessary complexity.

2. Criticality makes the final outcome of a test two dimensional so that one axis is the actual status and the other is criticality. Even with only pass and fail statuses we end up with four different end results "critical pass", "critical fail", "non-critical pass" and "non-critical fail", and adding the skip status to the mix would add "critical skip" and "non-critical skip". Most of these final statuses make no sense and everything is a lot easier if there's only "pass", "fail" and "skip".

3. When looking at suite statistics in reports and logs, Robot only shows the number of all passed and failed tests without any indication are failures critical or not. We have experimented showing statistics separately both for critical and non-critical tests but that didn't work well at all. This is similar problem as the one above and having just pass, fail and skip statuses resolved this one as well.

4. Criticality is a unique feature in Robot Framework. Unique tool features can be really useful, but they also require learning by new (and old) users and they don't always play nicely together with other tools. In this particular case skip is a familiar feature for most people working with automation and it's also a functionality that external tools like test management systems generally support.

5. Related to the above, having statistics both for "Critical Tests" and "All Tests" in reports and logs is rather strange especially for new users. Just having single statistics with pass, skip and fail statuses is a lot simpler and intuitive.

Migrating from criticality to skipping

Part of the new skip feature (#3622) is adding --skiponfailure command line option that automatically changes status of tests having a matching tag to skip if they fail. This works very much like the current --noncritical option that marks tests non-critical and thus their failures are in practice ignored. To make migration to skipping easier, --noncritical and also --critical will be preserved as aliases to --skiponfailure in RF 4.0.

Although --noncritical and --critical will continued to work mostly like earlier, there are various other changes affecting the current criticality users. Especially visible are changes in reports and logs where critical/non-critical distinction will be gone. It will, however, be easy to see statistics for all tests having a critical or non-critical tag in the tag statistics table. Other changes include removing the critical attribute from test elements in output.xml and changes to the result related APIs.

Migrating to skipping very importantly requires changes to integration with external tools. This will certainly add some work to projects providing such integration (e.g. Robot Framework Jenkins Plugin), but in the end using commonly used skip status and not the unique criticality is likely to make things easier.