Skip to content

BUG: Rebot --merge doesn't merge properly the reran results #4662

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
kilimandzsaro opened this issue Feb 17, 2023 · 4 comments
Open

BUG: Rebot --merge doesn't merge properly the reran results #4662

kilimandzsaro opened this issue Feb 17, 2023 · 4 comments

Comments

@kilimandzsaro
Copy link

I'm using the following commands for first run the tests in parallel, then rerun the failed tests and finally combine the results. Everything is running on docker machines, but I have the same result with rebot when I run them locally (on my mac).

In this example I have 2 failed tests for the first run
The second run executes only those 2 failed tests where only one of them is passing
When I combine the reports (using rebot with --merge), I can see 3 failed tests.

  • The Docker image is using python:3.11
  • Robotframework and Rebot version is 6.0.2
  • Pabot version is 2.13.0
  • My local machine is: MacBook Pro with Ventura 13.0

Commands I'm running and it's results (I won't upload the logs, because it contains information which I won't share, but here are screenshots):
1.

pipenv run pabot --processes 2 \
      --loglevel DEBUG \
      --exclude BLOCKED \
      --outputdir ${OUTPUT}/run \
      tests

Screenshot 2023-02-17 at 11 07 26

Screenshot 2023-02-17 at 11 09 15

pipenv run robot \
          --rerunfailed ${OUTPUT}/run/output.xml \
          --outputdir ${OUTPUT}/rerun/ \
          --loglevel DEBUG \
          --exclude BLOCKED \
          tests

Screenshot 2023-02-17 at 11 10 09

Screenshot 2023-02-17 at 11 10 21

pipenv run rebot \
          --outputdir ${OUTPUT} \
          --merge \
          --output merged_output.xml \
          ${OUTPUT}/run/pabot_results/*/output.xml ${OUTPUT}/rerun/output.xml

Screenshot 2023-02-17 at 11 11 04

Screenshot 2023-02-17 at 11 11 21

What I would expect from the combined report:
1 failure only (not 3)

I'm using rebot for a while and in the past the --merge was working as I expected. It started to acting up in the past month, since I updated my environment from python 3.8 to 3.11 and from robotframework 5.0.1 to 6.0.2
@pekkaklarck
Copy link
Member

Does this issue happen without Pabot as well? Does the exact same scenario work as you expect if you downgrade Robot to 5.0.1? It's very unlikely that Python version would matter.

@kilimandzsaro
Copy link
Author

Hi, I tried a lot of different things and I still don't know what can be the problem.

  1. I used the same output files with different version of robotframework and python and I had the same issue every time (python11 with robot 4.1.2, 5.0.0, 5.0.1, 6.0.2, python8 with robot 4.1.3, 5.0.0., 5.0.1, 6.0.2)
  2. Then I thought maybe if I rerun the whole tests (I have some which suppose to fail all the time) the same way as it runs in our pipeline I can get some results. I used for this python11, but I tried robot 5.0.1 and 6.0.2 multiple time. Every time the merge had an issue with the same test (it didn't merge that test's results). I checked the output.xmls of both the run and rerun, but there was no obvious difference there at all.
    Screenshot 2023-02-24 at 09 23 34
  3. The I though to run only that one test using which the merge had the issue on 6.0.2. Surprisingly the merge was a success.
    Screenshot 2023-02-24 at 09 24 34

I really don't know why the merge doesn't work properly all the time.

@oscar3037
Copy link

oscar3037 commented Apr 15, 2024 •
edited
Loading

I'm also seeing this issue when running the following command:
rebot --output .\final_output.xml --removekeywords passed --log _ATLOG.html --report NONE --merge .\*.xml
Windows 10
Python 3.9.6
robotframework: 7.0.0
robotframework-pabot: 2.18.0

This happens with and without pabot.

@pekkaklarck
Copy link
Member

Could someone create a simple example demonstrating the problem with Robot only?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pekkaklarck @kilimandzsaro @oscar3037