Skip to content

chore: skip some flaky tests #1643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 20, 2022
Merged

chore: skip some flaky tests #1643

merged 3 commits into from
May 20, 2022

Conversation

johnstcn
Copy link
Member

I ran gotestsum a few times and skipped some tests that flaked out.
This is not a long-term solution. We still need to fix these.

@johnstcn johnstcn requested a review from a team May 20, 2022 21:01
AbhineetJain
AbhineetJain approved these changes May 20, 2022
View reviewed changes
cli/ssh_test.go
@@ -23,6 +23,7 @@ import (
)

func TestSSH(t *testing.T) {
t.Skip("This is causing test flakes. TODO @cian fix this")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an issue filed for this? Maybe link it in a comment?

johnstcn
johnstcn commented May 20, 2022
View reviewed changes
@greyscaled
Copy link
Contributor

It looks like the test/go/postgres check is flaking out in this PR too 😢 . I'm not sure if the suppression here were for that or other tests. I hope we can use a similar tactic to hunt down the flakes in that check, or at least only run it when the data layer changes.

postgres

@johnstcn
Copy link
Member Author

It looks like the test/go/postgres check is flaking out in this PR too 😢

make test is not the same as what gets run in test/go/postgres!

@johnstcn
Copy link
Member Author

When I ran the postgres tests locally, they were working fine with a postgresql service running via Homebrew.
After I tried running the postgresql server in Docker, however, it ended up crashing at one point with this error:

2022-05-20 22:08:45.386 UTC [2168] LOG:  unexpected EOF on client connection with an open transaction
2022-05-20 22:08:45.439 UTC [1] LOG:  checkpointer process (PID 70) was terminated by signal 9: Killed
2022-05-20 22:08:45.439 UTC [1] LOG:  terminating any other active server processes
2022-05-20 22:08:45.439 UTC [2172] WARNING:  terminating connection because of crash of another server process
2022-05-20 22:08:45.439 UTC [2172] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-20 22:08:45.439 UTC [2172] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-20 22:08:45.439 UTC [2169] WARNING:  terminating connection because of crash of another server process
2022-05-20 22:08:45.439 UTC [2169] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-20 22:08:45.439 UTC [2169] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-20 22:08:45.440 UTC [73] WARNING:  terminating connection because of crash of another server process
2022-05-20 22:08:45.440 UTC [73] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-20 22:08:45.440 UTC [73] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-20 22:08:45.441 UTC [2173] WARNING:  terminating connection because of crash of another server process
2022-05-20 22:08:45.441 UTC [2173] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-20 22:08:45.441 UTC [2173] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-20 22:08:45.472 UTC [1] LOG:  all server processes terminated; reinitializing

@johnstcn
Copy link
Member Author

johnstcn commented May 20, 2022 •
edited
Loading

I'm experimenting reducing the level of parallelism in the postgres tests.
I'm finding that it's surprisingly easy for postgres to get OOM-killed when you run lots of tests at once. I think I saw the number of postgres processes peak at 137 or so (!)

@johnstcn
Copy link
Member Author

I also added some Makefile targets so we can more easily run the full-fat test suite and dig into these sorts of things locally.

@johnstcn johnstcn merged commit a8a8f9d into main May 20, 2022
@johnstcn johnstcn deleted the cj/test-flake-skip branch May 20, 2022 23:39
kylecarbs pushed a commit that referenced this pull request Jun 10, 2022
@johnstcn @kylecarbs
chore: skip some flaky tests (#1643)
1ac99d4 
* chore: skip some flaky tests

* Update peer/conn_test.go

* add makefile targets, reduce parallelism in go test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AbhineetJain AbhineetJain AbhineetJain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@johnstcn @greyscaled @AbhineetJain