Skip to content

How can I modify the summary with test results after testing has finished? #533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nck974 opened this issue Jul 18, 2022 · 9 comments · Fixed by #660
Closed

How can I modify the summary with test results after testing has finished? #533

nck974 opened this issue Jul 18, 2022 · 9 comments · Fixed by #660
Labels
next-gen Candidate for the next-gen project

Comments

@nck974
Copy link

nck974 commented Jul 18, 2022

I have been able to modify the summary with the following snippet from another issue:

def pytest_html_results_summary(prefix, summary, postfix):
    prefix.extend([
        html.button("Click me", onclick="myFunction()"),
        html.p(id="demo"),
        html.script(raw("""
        function myFunction() {
            document.getElementById('demo').innerHTML = 'Hello World';
        }""")),
    ])

And I am also able to gather the results after the session finishes with:

def pytest_sessionstart(session):
    """
    This hook is called at the beginning and creates a placeholder to store all the results of the
    session
    """
    session.results = dict()

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item):
    """
    Write the results of each test in the session
    """
    outcome = yield

    report = outcome.get_result()
    if report.when == "call":

        # Save result for later processing in pytest_sessionfinish hook
        item.session.results[item] = report

@pytest.hookimpl(tryfirst=True)
def pytest_sessionfinish(session, exitstatus):
    """
    This hook runs after all tests are done
    """
    print()
    print('run status code:', exitstatus)
    passed_amount = sum(1 for result in session.results.values() if result.passed)
    failed_amount = sum(1 for result in session.results.values() if result.failed)
    print(f'there are {passed_amount} passed and {failed_amount} failed tests')
    # TODO: Writing this in a file with a link

My problem is that I can not make a connection to write what I have in session finish with what I have in the summary. The summary hook seems to don't have have access to anyinformation of the session.

Furthermore, after making some prints, it seems the summary is generated before executing the pytest_sessionfinish.

Is there a work around to make my own summary additions after finishing the test?

@nck974
Copy link
Author

nck974 commented Jul 18, 2022

Well I was able to derive an ugly and hacky solution which I don't think is not the solution to the problem but is enough for me to be able to conitinue.

I was expecting to generate a button in the report that downloads an xml which is formatted as an import for another tool.

I just read the xml file and add at the end my button

@pytest.hookimpl(trylast=True, )
def pytest_sessionfinish(session, exitstatus):
    """
	There seems to be no way of using a hook of pytest-html to provide this summary results, 
	therefore the report is read once created
    and a custom HTML is added with a button which triggers a JS that downloads the XML.
    """
    # Get execution results
    test_cases = {}
    for item, result in session.results.items():

        try:
            tc = re.search(r'^(.+?)_test.py', item.parent.name).group(1)
        except AttributeError:
            logger.error(f'tc not found in module {item.parent.name}')
            raise
        
        # Passed
        if result.passed:
            if tc not in test_cases:
                test_cases[tc] = 'p'
            elif test_cases[tc] == 'f':
                continue
            else:
                test_cases[tc] = 'p'
        
        # Failed
        if result.failed:
            test_cases[tc] = 'f'

    # Generate xml file
	xml = 'x'


    # Get name of the generated report
    output = os.path.join(
        'output',
        'report',
        'my_project',
        datetime.now().strftime("%Y-%m-%d"),
        f'{MY_PROJECT}-test-report.html')

    # Read report
    with open(output) as f:
        lines = f.read()
    
    button = f"""
    <button onclick="downloadBase64File('text/xml','{base64.b64encode(xml.encode('utf8')).decode()}', 'test.xml' )">
    Button to download xml
    </button>
    <span style="color: #756e15;background: #fffbd1;border: 1px solid #87803e;margin: 5px;padding: 5px;" >
        Warning: Add here.
    </span>
    <script>
            function downloadBase64File(contentType, base64Data, fileName) {{
            const linkSource = `data:${{contentType}};base64,${{base64Data}}`;
            const downloadLink = document.createElement("a");
            downloadLink.href = linkSource;
            downloadLink.download = fileName;
            downloadLink.click();
        }}
    </script>
    """
    lines = re.sub(re.escape(
        '</body></html>'),
        f'<div style="padding: 5px;">{button}</div></body></html>',
        lines)
    
    # Save again the report with the same name
    with open(output, 'w') as f:
        f.write(lines)

And looks likes this:
image

@BeyondEvil
Copy link
Contributor

Not sure what the correct solution is.

But let me label this next-gen, and once we've figured out how to deal with the hooks that modify the report - we can look at this use case.

@BeyondEvil BeyondEvil added the next-gen Candidate for the next-gen project label Jul 18, 2022
@BeyondEvil
Copy link
Contributor

Please try 4.0.0rc0 to see if this still is an issue.

@nck974
Copy link
Author

nck974 commented Mar 10, 2023

I'm not sure if this is an issue. As I commented before I used that hacky solution of writing over the HTML, if a hook can be implemented to be able to what is described in the title would be nice. Otherwise I do not have the need right now to change it.

@BeyondEvil
Copy link
Contributor

BeyondEvil commented Mar 10, 2023

I'm not sure if this is an issue. As I commented before I used that hacky solution of writing over the HTML, if a hook can be implemented to be able to what is described in the title would be nice. Otherwise I do not have the need right now to change it.

Not 100% sure I understand the problem.

But here's what the current (4.0.0rc0) implementation looks like:

    @pytest.hookimpl(trylast=True)
    def pytest_sessionfinish(self, session):
        session.config.hook.pytest_html_results_summary(
            prefix=self._report.data["additionalSummary"]["prefix"],
            summary=self._report.data["additionalSummary"]["summary"],
            postfix=self._report.data["additionalSummary"]["postfix"],
        )
        self._report.data["runningState"] = "Finished"
        self._generate_report()

Would changing that to something like:

    @pytest.hookimpl(trylast=True)
    def pytest_sessionfinish(self, session):
        session.config.hook.pytest_html_results_summary(
            prefix=self._report.data["additionalSummary"]["prefix"],
            summary=self._report.data["additionalSummary"]["summary"],
            postfix=self._report.data["additionalSummary"]["postfix"],
            session=session,
        )
        self._report.data["runningState"] = "Finished"
        self._generate_report()

solve your issue?

@nck974
Copy link
Author

nck974 commented Mar 10, 2023

At first glance I think it would enough. Unfortunately I won't have time to verify that any time soon, as I do not have quick access right now to the code where I was using that.

@BeyondEvil
Copy link
Contributor

BeyondEvil commented Mar 10, 2023

At first glance I think it would enough. Unfortunately I won't have time to verify that any time soon, as I do not have quick access right now to the code where I was using that.

No worries. There's no downside afaict to adding the session info to the hook. I'll make it part of the next RC so you can test whenever you have time.

@nahuelmariani
Copy link

Hi @BeyondEvil! I'm facing the same situation described by @nck974. I would love to have the session as parameter of the pytest_html_results_summary hook in the new RC.
Thank you so much!

@BeyondEvil
Copy link
Contributor

Thanks for reminding me! I’ll see if I can get it added today and release a new RC you can try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
next-gen Candidate for the next-gen project
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants