Skip to content

Introduce Codecov's Test Analysis #4487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Sep 21, 2024
Merged

Introduce Codecov's Test Analysis #4487

merged 3 commits into from
Sep 21, 2024

Conversation

Bibo-Joshi
Copy link
Member

See https://docs.codecov.com/docs/test-result-ingestion-beta and https://about.codecov.io/blog/find-failing-and-flaky-tests-with-codecov-test-analytics/

In light of the discussion in #4324 and related threads, it seemed like a good idea to try it out :)

@Bibo-Joshi Bibo-Joshi added the ⚙️ tests affected functionality: tests label Sep 19, 2024
@codecov Codecov
Copy link

codecov bot commented Sep 19, 2024 •
edited
Loading

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
5995 1 5994 251
View the top 1 failed tests by shortest run time 
tests.test_bot.TestBotWithRequest test_copy_messages
Stack Traces | 0.623s run time 
self = <tests.test_bot.TestBotWithRequest object at 0x1095ac5f0>
bot = PytestExtBot[token=5744644522:AAEAdsrF0hC6p6HUO0P01Q8b_jChU52XG-M]
chat_id = '675666224'

    async def test_copy_messages(self, bot, chat_id):
        tasks = asyncio.gather(
            bot.send_message(chat_id, text="will be copied 1"),
            bot.send_message(chat_id, text="will be copied 2"),
        )
        msg1, msg2 = await tasks
    
        copy_messages = await bot.copy_messages(
            chat_id, from_chat_id=chat_id, message_ids=sorted((msg1.message_id, msg2.message_id))
        )
        assert isinstance(copy_messages, tuple)
    
        tasks = asyncio.gather(
            bot.send_message(chat_id, "temp 1", reply_to_message_id=copy_messages[0].message_id),
            bot.send_message(chat_id, "temp 2", reply_to_message_id=copy_messages[1].message_id),
        )
        temp_msg1, temp_msg2 = await tasks
    
        forward_msg1 = temp_msg1.reply_to_message
        forward_msg2 = temp_msg2.reply_to_message
    
>       assert forward_msg1.text == msg1.text
E       AssertionError: assert 'will be copied 2' == 'will be copied 1'
E         
E         - will be copied 1
E         ?                ^
E         + will be copied 2
E         ?                ^

tests/test_bot.py:3977: AssertionError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

@Bibo-Joshi
Copy link
Member Author

The PR comment is a downside IMO but one that I can live with. I've already pointed out that I'd like to have an option to disable it: codecov/feedback#304 (comment)

harshil21
harshil21 reviewed Sep 20, 2024
View reviewed changes
Copy link
Member

@harshil21 harshil21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really like the codecov overview of the tests, test durations is something we can work on directly, and failure rate is cumulative, an insight which is hard to gather using local pytest runs.

@Bibo-Joshi Bibo-Joshi merged commit 3409f51 into master Sep 21, 2024
25 of 26 checks passed
@Bibo-Joshi Bibo-Joshi deleted the codecov-test-analysis branch September 21, 2024 15:15
@github-actions github-actions bot locked and limited conversation to collaborators Sep 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@harshil21 harshil21 harshil21 approved these changes

Assignees
No one assigned
Labels
⚙️ tests affected functionality: tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Bibo-Joshi @harshil21