Skip to content

Execution in teardown should continue after keyword timeout #3398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
juhoarvid opened this issue Dec 2, 2019 · 13 comments
Closed

Execution in teardown should continue after keyword timeout #3398

juhoarvid opened this issue Dec 2, 2019 · 13 comments

Comments

@juhoarvid
Copy link

Version information
Robot Framework version: 3.1.2
Python interpreter type Python and version: Python 2.7.16
Operating system and its version: Centos 7:3.10.0-1062.4.1.el7.x86_64
Steps to reproduce the problem.
See attachment
teardown_not_all_kw_executed.zip

  • If there is timeout in teardown sub keyword and keyword fails due to that, no other keyword will be executed, even if there is no timeouts on those

EXPECTED RESULT:
https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#test-setup-and-teardown
"
In addition, all the keywords in the teardown are also executed even if one of them fails. This continue on failure functionality can be used also with normal keywords, but inside teardowns it is on by default.
"

@pekkaklarck
Copy link
Member

Could you please explain the problem a bit more? I'm on mobile and cannot open the zip file.

@juhoarvid
Copy link
Author

juhoarvid commented Dec 2, 2019

If test teardown is and keyword and if any of it's sub keywords has timeout and fails due to timeout, no other sub keywords will be executed.

Here is the same in text format, quite short:

*** Keywords ***
Teardown Keyword With Timeout Fail
    Log To Console  TEARDOWN START
    Failing Kw Due To Timeout
    Passing KW
    Log To Console  TEARDOWN ENDS

Teardown Keyword With Fail
    Log To Console  TEARDOWN START
    Failing Kw
    Passing KW
    Log To Console  TEARDOWN ENDS

Failing Kw Due To Timeout
    [Timeout]  1 sec
    Sleep  2

Failing Kw
    Fail  This keyword allways fails

Passing KW
    Log  I am always doing cleanup well and going pass

*** Test Cases ***
Teardown Continues On Failure
    [Teardown]  Teardown Keyword With Fail
    Log  Test executed

Teardown Stop In keyword Timeout
    [Teardown]  Teardown Keyword With Timeout Fail
    Log  Test executed

@pekkaklarck
Copy link
Member

That's by design and ought to also be documented.

@pekkaklarck
Copy link
Member

Now that I think about this more, I guess it would be fine to continue teardown after that keyword that has failed due to a timeout. Is someone interested to study how big a change it would be? I probably won't have time for that before RF 3.2.

@pekkaklarck pekkaklarck changed the title Test Teardown "continue on failure" feature not working if sub keyword reach timeout Execution in teardown should continue after keyword timeout Dec 3, 2019
@stdedos
Copy link
Contributor

stdedos commented Dec 3, 2019

https://robotframework.slack.com/archives/C3C28F9DF/p1572513770127500

Test Teardown
    Keyword 1
        # Which contains:
        Keyword 1.1
        Keyword 1.2
        Keyword 1.3
        Keyword 1.4
            # Which contains:
            Keyword 1.4.1 # <--- This one times out
        Keyword 1.5

    Keyword 2
        # Which contains:
        Keyword 2.1


    Keyword 3
        # Which contains:
        Keyword 3.1


    Keyword 4
        # Which contains:
        Keyword 4.1

Keyword 1.4.1 times out, but Keyword 1.5, Keyword 2 and the rest are not executed

@pekkaklarck
Copy link
Member

Looked at this quickly and noticed this might be very easy to fix. These changes seem to be enough:

--- a/src/robot/errors.py
+++ b/src/robot/errors.py
@@ -146,10 +146,10 @@ class ExecutionStatus(RobotError):
             return False
         if templated:
             return True
-        if self.keyword_timeout:
-            return False
         if teardown:
             return True
+        if self.keyword_timeout:
+            return False

Could someone encountering this in real environment test the above changes? If the fix works, and there are no problems, this could be included in RF 3.2. Some tests would obviously be needed but adding them shouldn't be overly complicated. I can do them myself, but if someone wants to create a PR that would be even better.

@pekkaklarck
Copy link
Member

I executed Robot's acceptance tests with the above changes and the only test that failed explicitly verified that execution in teardown would stop after a timeout. That's great news, because now we know there are tests for this feature and just update them to match the new design and possibly add some new ones. Should also check what the User Guide says about this.

@juhoarvid
Copy link
Author

Looked at this quickly and noticed this might be very easy to fix. These changes seem to be enough:

--- a/src/robot/errors.py
+++ b/src/robot/errors.py
@@ -146,10 +146,10 @@ class ExecutionStatus(RobotError):
             return False
         if templated:
             return True
-        if self.keyword_timeout:
-            return False
         if teardown:
             return True
+        if self.keyword_timeout:
+            return False

Could someone encountering this in real environment test the above changes? If the fix works, and there are no problems, this could be included in RF 3.2. Some tests would obviously be needed but adding them shouldn't be overly complicated. I can do them myself, but if someone wants to create a PR that would be even better.

I just wanted to add comment to ensure there is still ability to add timeout to teardown itself. For instance some of keyword are jumming. But tested your change request with following code and seems it fails due to timeout after 10 sec of executing 'TDKW'

E.g.


*** Settings ***
Test Teardown  TDKW

*** Keywords ***
TDKW
  [Timeout]  10
  SubTDKW1
  SubTDKW2

SubTDKW1
  [Timeout]  4
  Sleep  5

SubTDKW2
  #JUMMING KW
  Sleep  1000

*** Test Cases ***
Test Teardown Timeouts
   Log  Test

@pekkaklarck
Copy link
Member

Yeah, you should still be able to add a timeout to the top level teardown keyword to stop its execution for good. With the above simple change that doesn't work fully, though. If you have, for example, a keyword like the one below, keywords after the point where the timeout occurs won't be executed (i.e. the example won't fail with "Should not be executed"). Unfortunately those keywords are run so that they timeout, so the teardown actually fails four times with "Keyword timeout 100 milliseconds exceeded." message. That's not acceptable. If a keyword fails for timeout, remaining keywords in it should be skipped.

*** Keywords ***
Example
    [Timeout]    0.1 seconds
    Sleep    1 second
    Fail    Should not be executed
    Fail    Should not be executed
    Fail    Should not be executed

@stdedos
Copy link
Contributor

stdedos commented Dec 4, 2019

^^ sounds like a prime candidate for a negative test

@AlexanderShikov
Copy link

Any news on this one? We are experiencing this issue in a real environment, with teardown not completing fully.
RF version 3.1.2

@pekkaklarck
Copy link
Member

Let's see could we get this done in RF 4.1.

@pekkaklarck pekkaklarck added this to the v4.1 milestone Mar 3, 2021
@pekkaklarck
Copy link
Member

The above commit changes the logic so that keyword where the timeout occurs is stopped but other keywords can continue execution. Would be great if someone who has encountered this issue could validate the behavior in their environment. There's going to be RF 4.1 alpha or beta release somewhat soon, but you can already before that install the latest code directly from GitHub is you want.

pekkaklarck added a commit that referenced this issue Jun 24, 2021
Code in master was changed to fix #3398 before merging #3925 in a way
that caused this failure.
pekkaklarck added a commit that referenced this issue Oct 18, 2024
Turning keyword timeouts that occurred in teardowns to normal errors
is needed as a fix for #3398. Better to do that explicitly than as
a side-effect of `can_continue` call.
pekkaklarck added a commit that referenced this issue Oct 18, 2024
pekkaklarck added a commit that referenced this issue Oct 18, 2024
Turning keyword timeouts that occurred in teardowns to normal errors
is needed as a fix for #3398. Better to do that explicitly than as
a side-effect of `can_continue` call.
pekkaklarck added a commit that referenced this issue Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants