Skip to content

Enable continuable failures via robot:continue-on-failure test or kw tag #3925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 43 commits into from
Jun 24, 2021
Merged

Enable continuable failures via robot:continue-on-failure test or kw tag #3925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
a2d666c
allow test case to continue on failure based on tag
oboehmer Apr 7, 2021
dc8f9d7
fix test to check for log
oboehmer Apr 7, 2021
d2ccc34
remove redundant var assignment
oboehmer Apr 7, 2021
7619435
minor cleanup
oboehmer Apr 7, 2021
2d129ce
added robot:continue-on-failure for loops
oboehmer Apr 7, 2021
51fe90a
add tags attribute to BodyRunner object, this fixes keyword tags
oboehmer Apr 7, 2021
0552cd3
move tests into new suite
oboehmer Apr 7, 2021
5a27dda
merge test and user kw tags before run
oboehmer Apr 7, 2021
4c1c127
add _tags to ForInRunner object used for continue-on-failure handling
oboehmer Apr 7, 2021
d99306e
cosmetic cleanup
oboehmer Apr 7, 2021
89bde5b
additional test cases for for loop
oboehmer Apr 7, 2021
9c23912
added more test cases
oboehmer Apr 8, 2021
5b4c052
allow continuable failures to continue even if robot:no-continue-on-f…
oboehmer Apr 8, 2021
eb3690c
add documentation
oboehmer Apr 8, 2021
531964a
Merge branch 'master' into 2285-continue-tag
oboehmer Apr 8, 2021
0f16b72
Merge remote-tracking branch 'upstream/master' into 2285-continue-tag
oboehmer Apr 9, 2021
d4a24ea
align ForInRunner signature to Bodyrunner's
oboehmer Apr 9, 2021
f3b2f48
cosmetic changes to tests
oboehmer Apr 9, 2021
ebe9f5a
minor doc correction
oboehmer Apr 10, 2021
c9453cd
another minor doc correction
oboehmer Apr 10, 2021
6878199
reformat according to black guideline
oboehmer Apr 11, 2021
e37d4ad
documented the behaviour that no-continue-on-failure does not change …
oboehmer Apr 11, 2021
0293f8c
Merge branch 'master' into 2285-continue-tag
oboehmer May 31, 2021
d59aa9e
cosmetic/naming changes
oboehmer May 31, 2021
e1e1d3e
Merge branch 'master' into 2285-continue-tag
oboehmer Jun 8, 2021
26c3166
Address review comments
oboehmer Jun 21, 2021
84573ee
updated docs
oboehmer Jun 21, 2021
9c0e7f2
Merge branch '2285-continue-tag' of github.com:oboehmer/robotframewor…
oboehmer Jun 21, 2021
c5952a8
refactor implementation to leverage context instead of tags in Runners
oboehmer Jun 21, 2021
03ca263
change user_keyword signature
oboehmer Jun 21, 2021
a16fb7f
simplify errors.can_continue() signature
oboehmer Jun 22, 2021
4f3bb73
change var name in continue_on_failure
oboehmer Jun 22, 2021
b67c780
introduce recursive behaviour with dedicated -recursive tag
oboehmer Jun 22, 2021
c13d1cf
suppress internal tags to not show by default
oboehmer Jun 22, 2021
26130f7
add integration tests and docs for internal tag suppression
oboehmer Jun 23, 2021
9b3f83b
cosmetic naming change of suppress logic
oboehmer Jun 23, 2021
49d1d5b
document recursive behaviour
oboehmer Jun 23, 2021
b72ad12
add tests to verify set tags functionality
oboehmer Jun 23, 2021
4bc2f88
test commit
oboehmer Jun 23, 2021
36902f5
test changes as per review comments
oboehmer Jun 23, 2021
c27f3cb
doc changes as per review comments
oboehmer Jun 23, 2021
c7a1491
minor code changes as per review comments
oboehmer Jun 23, 2021
7013791
reverted back to non-dash notation in doc
oboehmer Jun 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions atest/robot/running/continue_on_failure_tag.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
*** Settings ***
Suite Setup Run Tests ${EMPTY} running/continue_on_failure_tag.robot
Resource atest_resource.robot

*** Test Cases ***
Continue in test with tag
Check Test Case ${TESTNAME}

Continue in test with Set Tags
Check Test Case ${TESTNAME}

Continue in user keyword with tag
Check Test Case ${TESTNAME}

Continue in test with tag and UK without tag
Check Test Case ${TESTNAME}

Continue in test with tag and nested UK with and without tag
Check Test Case ${TESTNAME}

Continue in test with tag and two nested UK with tag
Check Test Case ${TESTNAME}

Continue in FOR loop with tag
Check Test Case ${TESTNAME}

Continue in FOR loop with Set Tags
Check Test Case ${TESTNAME}

No continue in FOR loop without tag
Check Test Case ${TESTNAME}

Continue in FOR loop in UK with tag
Check Test Case ${TESTNAME}

Continue in FOR loop in UK without tag
Check Test Case ${TESTNAME}

Continue in IF with tag
Check Test Case ${TESTNAME}

Continue in IF with set and remove tag
Check Test Case ${TESTNAME}

No continue in IF without tag
Check Test Case ${TESTNAME}

Continue in IF in UK with tag
Check Test Case ${TESTNAME}

No continue in IF in UK without tag
Check Test Case ${TESTNAME}

Continue in Run Keywords with tag
Check Test Case ${TESTNAME}

Recursive continue in test with tag and two nested UK without tag
Check Test Case ${TESTNAME}

Recursive continue in test with Set Tags and two nested UK without tag
Check Test Case ${TESTNAME}

Recursive continue in test with tag and two nested UK with and without tag
Check Test Case ${TESTNAME}

Recursive continue in user keyword
Check Test Case ${TESTNAME}

No recursive continue in user keyword
Check Test Case ${TESTNAME}
9 changes: 9 additions & 0 deletions atest/robot/tags/tag_stat_include_and_exclude.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ Resource atest_resource.robot
*** Variables ***
${DATA SOURCE} tags/include_and_exclude.robot
${F} force
${INTERNAL} robot:just-an-example
${I1} incl1
${I2} incl 2
${I3} incl_3
Expand Down Expand Up @@ -32,6 +33,14 @@ Include With Patterns
--TagStatInc incl_? @{INCL}
--TagStatInc *cl3 --TagStatInc i*2 ${E3} ${I2} ${I3}

Include to show internal tags
--tagstatinclude incl1 --tagstatinclude robot:* ${I1} ${INTERNAL}
--tagstatinclude robot:* ${INTERNAL}
--tagstatinclude * @{ALL} ${INTERNAL}

Include and exclude internal
--tagstatinclude incl1 --tagstatinclude robot:* --tagstatexclude robot:* ${I1}

One Exclude
--tagstatexclude excl1 ${E2} ${E3} ${F} @{INCL}

Expand Down
9 changes: 9 additions & 0 deletions atest/robot/tags/tag_stat_include_and_exclude_with_rebot.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Resource rebot_resource.robot
${DATA SOURCE} tags/include_and_exclude.robot
${INPUT FILE} %{TEMPDIR}${/}robot-test-tagstat.xml
${F} force
${INTERNAL} robot:just-an-example
${I1} incl1
${I2} incl 2
${I3} incl_3
Expand Down Expand Up @@ -35,6 +36,14 @@ Include With Patterns
--TagStatInc incl_? @{INCL}
--TagStatInc *cl3 --TagStatInc i*2 ${E3} ${I2} ${I3}

Include to show internal tags
--tagstatinclude incl1 --tagstatinclude robot:* ${I1} ${INTERNAL}
--tagstatinclude robot:* ${INTERNAL}
--tagstatinclude * @{ALL} ${INTERNAL}

Include and exclude internal
--tagstatinclude incl1 --tagstatinclude robot:* --tagstatexclude robot:* ${I1}

One Exclude
--tagstatexclude excl1 ${E2} ${E3} ${F} @{INCL}

Expand Down
263 changes: 263 additions & 0 deletions atest/testdata/running/continue_on_failure_tag.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,263 @@
*** Variables ***
${HEADER} Several failures occurred:

*** Test Cases ***
Continue in test with tag
[Documentation] FAIL ${HEADER}\n\n
... 1) 1\n\n
... 2) 2
[Tags] robot:continue-on-failure
Fail 1
Fail 2
Log This should be executed

Continue in test with Set Tags
[Documentation] FAIL ${HEADER}\n\n
... 1) 1\n\n
... 2) 2
Set Tags robot:continue-on-failure
Fail 1
Fail 2
Log This should be executed

Continue in user keyword with tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw1a\n\n
... 2) kw1b
Failure in user keyword with tag
Fail This should not be executed

Continue in test with tag and UK without tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw2a\n\n
... 2) This should be executed
[Tags] robot:continue-on-failure
Failure in user keyword without tag
Fail This should be executed

Continue in test with tag and nested UK with and without tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw1a\n\n
... 2) kw1b\n\n
... 3) kw2a\n\n
... 4) This should be executed
[Tags] robot:continue-on-failure
Failure in user keyword with tag run_kw=Failure in user keyword without tag
Fail This should be executed

Continue in test with tag and two nested UK with tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw1a\n\n
... 2) kw1b\n\n
... 3) kw1a\n\n
... 4) kw1b\n\n
... 5) This should be executed
[Tags] robot:continue-on-failure
Failure in user keyword with tag run_kw=Failure in user keyword with tag
Fail This should be executed

Continue in FOR loop with tag
[Documentation] FAIL ${HEADER}\n\n
... 1) loop-1\n\n
... 2) loop-2\n\n
... 3) loop-3
[Tags] robot:continue-on-failure
FOR ${val} IN 1 2 3
Fail loop-${val}
END

Continue in FOR loop with Set Tags
[Documentation] FAIL ${HEADER}\n\n
... 1) loop-1\n\n
... 2) loop-2\n\n
... 3) loop-3
FOR ${val} IN 1 2 3
Set Tags robot:continue-on-failure
Fail loop-${val}
END

No continue in FOR loop without tag
[Documentation] FAIL loop-1
FOR ${val} IN 1 2 3
Fail loop-${val}
END

Continue in FOR loop in UK with tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw-loop-1\n\n
... 2) kw-loop-2\n\n
... 3) kw-loop-3
FOR loop in in user keyword with tag

Continue in FOR loop in UK without tag
[Documentation] FAIL kw-loop-1
FOR loop in in user keyword without tag

Continue in IF with tag
[Documentation] FAIL ${HEADER}\n\n
... 1) 1\n\n
... 2) 2\n\n
... 3) 3\n\n
... 4) 4
[Tags] robot:continue-on-failure
IF 1==1
Fail 1
Fail 2
END
IF 1==2
No Operation
ELSE
Fail 3
Fail 4
END

Continue in IF with set and remove tag
[Documentation] FAIL ${HEADER}\n\n
... 1) 1\n\n
... 2) 2\n\n
... 3) 3
Set Tags robot:continue-on-failure
IF 1==1
Fail 1
Fail 2
END
Remove Tags robot:continue-on-failure
IF 1==2
No Operation
ELSE
Fail 3
Fail this is not executed
END

No continue in IF without tag
[Documentation] FAIL 1
IF 1==1
Fail 1
Fail This should not be executed
END

Continue in IF in UK with tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw1a\n\n
... 2) kw1b\n\n
... 3) kw1c\n\n
... 4) kw1d
IF in user keyword with tag

No continue in IF in UK without tag
[Documentation] FAIL kw1a
IF in user keyword without tag

Continue in Run Keywords with tag
[Documentation] FAIL ${HEADER}\n\n
... 1) 1\n\n
... 2) 2
[Tags] robot:continue-on-failure
Run Keywords Fail 1 AND Fail 2

Recursive continue in test with tag and two nested UK without tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw2a\n\n
... 2) kw2b\n\n
... 3) kw2a\n\n
... 4) kw2b\n\n
... 5) This should be executed
[Tags] robot:continue-on-failure-recursive
Failure in user keyword without tag run_kw=Failure in user keyword without tag
Fail This should be executed

Recursive continue in test with Set Tags and two nested UK without tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw2a\n\n
... 2) kw2b\n\n
... 3) kw2a\n\n
... 4) kw2b\n\n
... 5) This should be executed
Set Tags robot:continue-on-failure-recursive
Failure in user keyword without tag run_kw=Failure in user keyword without tag
Fail This should be executed

Recursive continue in test with tag and two nested UK with and without tag
[Documentation] FAIL ${HEADER}\n\n
... 1) kw1a\n\n
... 2) kw1b\n\n
... 3) kw2a\n\n
... 4) kw2b\n\n
... 5) This should be executed
[Tags] robot:continue-on-failure-recursive
Failure in user keyword with tag run_kw=Failure in user keyword without tag
Fail This should be executed

Recursive continue in user keyword
[Documentation] FAIL ${HEADER}\n\n
... 1) kw1a\n\n
... 2) kw1b\n\n
... 3) kw2a\n\n
... 4) kw2b
Failure in user keyword with recursive tag run_kw=Failure in user keyword without tag
Fail This should not be executed

No recursive continue in user keyword
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this name accurate? Doesn't the test validate recursive continue works also with nested keywords?

[Documentation] FAIL kw2a
Failure in user keyword without tag run_kw=Failure in user keyword with recursive tag
Fail This should not be executed

*** Keywords ***

Failure in user keyword with tag
[Arguments] ${run_kw}=No Operation
[Tags] robot:continue-on-failure
Fail kw1a
Fail kw1b
Log This should be executed
Run Keyword ${run_kw}

Failure in user keyword without tag
[Arguments] ${run_kw}=No Operation
Fail kw2a
Fail kw2b
Run Keyword ${run_kw}

Failure in user keyword with recursive tag
[Arguments] ${run_kw}=No Operation
[Tags] robot:continue-on-failure-recursive
Fail kw1a
Fail kw1b
Log This should be executed
Run Keyword ${run_kw}

FOR loop in in user keyword with tag
[Tags] robot:continue-on-failure
FOR ${val} IN 1 2 3
Fail kw-loop-${val}
END

FOR loop in in user keyword without tag
FOR ${val} IN 1 2 3
Fail kw-loop-${val}
END

IF in user keyword with tag
[Tags] robot:continue-on-failure
IF 1==1
Fail kw1a
Fail kw1b
END
IF 1==2
No Operation
ELSE
Fail kw1c
Fail kw1d
END

IF in user keyword without tag
IF 1==1
Fail kw1a
Fail kw1b
END
IF 1==2
No Operation
ELSE
Fail kw1c
Fail kw1d
END
2 changes: 1 addition & 1 deletion atest/testdata/tags/include_and_exclude.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
*** Settings ***
Force Tags force
Force Tags force robot:just-an-example

*** Test Cases ***
Incl-1
Expand Down
Loading