-
Notifications
You must be signed in to change notification settings - Fork 253
Description
Environment:
Script: runner.py
Dependencies: Python (3.12.6),
allure-behave==2.14.0,
allure-python-commons==2.13.3 ,
behave==1.2.6
Platform: MAC OS
Summary: While executing a runner.py script using a for loop to iterate over a list of devices, I observe unexpected behavior in the generated Allure results. Specifically, for each iteration through device_list, the count of test cases reported by Allure results grows exponentially, despite running a single test for each device.
Description:
Code snippet:
for device in devices:
device_serial = (device.get("serial") if isinstance(device, dict) else device)
allure_device_results_dir, report_device_path = create_device_folders(str(PLATFORM.value), str(device_serial))
if os.path.exists("allure-results"):
shutil.rmtree("allure-results")
os.makedirs("allure-results", exist_ok=True)
runner.main(
f"./features/ -f allure_behave.formatter:AllureFormatter "
f"-o allure-results --no-capture --no-capture-stderr --no-skipped "
f"--tags={TAG} --tags=~@ignore "
f"--define board={BOARD.value} "
f"--define product={PRODUCT} --define solution={SOLUTION} "
f"--define device={device_serial} "
)
Observed Behaviour:
During first iteration (device 1), one result.json is produced, showing 1 test case count in Allure serve.
During second iteration (device 2), two result.json files are produced, showing 2 test cases count in Allure serve, although only one test occurs.
This pattern repeats for subsequent devices, with counts increasing exponentially.
Expected Behaviour:
For each device in the list, only one result.json should be created corresponding to the single test executed, reflecting accurate test case counts.
Steps to Reproduce:
Setup the script with a list of devices.
Execute runner.py to iterate over each device.
Observe the contents of allure-results directory and run allure serve.
Impact:
Incorrect test case count presentation.
Potential confusion during test result analysis.
Below is the screenshot of first iteration:
and below is the screenshot of 4th iteration: