Skip to content

skip flaky transcribe tests #12260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 13, 2025
Merged

skip flaky transcribe tests #12260

merged 1 commit into from
Feb 13, 2025

Conversation

alexrashed
Copy link
Member

Motivation

We have seen a bunch of flaky runs with the Transcribe tests in the past, and it seems that we still see issues.
This PR skips the last test which actually performs an actual transcription task. This should definitely be addressed fairly soon, but for the sake of the stability of our pipelines we are right now actively skipping all flaky tests.
/cc @sannya-singal

Changes

  • Skips the last transcribe service integration tests which actually perform transcriptions.

@alexrashed alexrashed added the semver: patch Non-breaking changes which can be included in patch releases label Feb 13, 2025
@alexrashed alexrashed added this to the 4.2 milestone Feb 13, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 13, 2025 •
edited
Loading

LocalStack Community integration with Pro

 2 files  ±    0  2 suites  ±0   37s ⏱️ - 1h 53m 16s
23 tests  - 4 074  6 ✅  - 3 775  17 💤  - 299  0 ❌ ±0 
25 runs   - 4 074  6 ✅  - 3 775  19 💤  - 299  0 ❌ ±0 

Results for commit 2ee393f. ± Comparison against base commit 25d4d82.

This pull request removes 4074 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…
This pull request skips 15 tests. 
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_happy_path
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[None-None]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-2-None]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-3-test-output]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-4-test-output.json]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-5-test-files/test-output.json]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-6-test-files/test-output]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.amr-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.flac-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.mp3-hello my name is]
…

♻️ This comment has been updated with latest results.

sannya-singal
sannya-singal approved these changes Feb 13, 2025
View reviewed changes
Copy link
Contributor

@sannya-singal sannya-singal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for jumping on this 🦸 LGTM!

@alexrashed alexrashed merged commit a79d200 into master Feb 13, 2025
23 of 28 checks passed
@alexrashed alexrashed deleted the fix-flaky-tests-2 branch February 13, 2025 09:28
tiurin added a commit that referenced this pull request Apr 3, 2025
@tiurin
Skip test_transcribe_start_job that failed on PR run
76dcfac 
It was also skipped before in #12260
@tiurin tiurin mentioned this pull request Apr 3, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ackdav ackdav ackdav approved these changes

@sannya-singal sannya-singal sannya-singal approved these changes

Assignees
No one assigned
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Milestone
4.2
Development

Successfully merging this pull request may close these issues.
3 participants
@alexrashed @ackdav @sannya-singal