Skip to content

fix: Fix flakeyness of TestProvisionerd/ReconnectAndComplete #5169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 24, 2022
Merged

fix: Fix flakeyness of TestProvisionerd/ReconnectAndComplete #5169

merged 1 commit into from
Nov 24, 2022

Conversation

mafredri
Copy link
Member

@mafredri mafredri commented Nov 24, 2022 •
edited
Loading

I noticed this sleep was added to another test in #5136 as well. Since TestProvisionerd/ReconnectAndComplete was constantly flaking, I added it there as well. It seems to have alleviated the issue but I don't really even understand this test.

Why is this issue triggered via memDRPC and not yamux? Maybe because memDRPC is faster? Or perhaps isn't handling some edge case in the same way causing "reconnect" to fail.

Logs on fail:

    t.go:81: 2022-11-24 11:37:34.016 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/runner/runner.go:333>	(*Runner).doCleanFinish.func2	cleaned up work directory	{"job_id": "test"}
    t.go:81: 2022-11-24 11:37:34.377 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/runner/runner.go:194>	(*Runner).Run	sending CompletedJob	{"job_id": "test"}
    t.go:81: 2022-11-24 11:37:34.429 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/provisionerd.go:200>	(*Server).connect.func1	client stream ended
    t.go:86: 2022-11-24 11:37:34.429 [ERROR]	(provisionerd)	<github.com/coder/coder/provisionerd/runner/runner.go:197>	(*Runner).Run	send CompletedJob	{"job_id": "test"} ...
        "error": connection closed
                 	storj.io/drpc/drpcstream.(*Stream).sendPacket:268
                 	storj.io/drpc/drpcstream.(*Stream).Close:478
                 	storj.io/drpc/drpcconn.(*Conn).Invoke.func1:80
                 	storj.io/drpc/drpcconn.(*Conn).Invoke:96
                 	github.com/coder/coder/provisionersdk.(*memDRPC).Invoke:103
                 	github.com/coder/coder/provisionerd/proto.(*drpcProvisionerDaemonClient).CompleteJob:96
                 	github.com/coder/coder/provisionerd.(*Server).CompleteJob.func1:406
                 	github.com/coder/coder/provisionerd.(*Server).clientDoWithRetries:366
                 	github.com/coder/coder/provisionerd.(*Server).CompleteJob:405
                 	github.com/coder/coder/provisionerd/runner.(*Runner).Run:195
    t.go:81: 2022-11-24 11:37:34.530 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/provisionerd.go:176>	(*Server).connect	connected
    t.go:81: 2022-11-24 11:37:34.584 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/provisionerd.go:489>	(*Server).closeWithError	closing server with error	{"error": null}

Logs on success:

    t.go:81: 2022-11-24 11:37:36.934 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/runner/runner.go:333>	(*Runner).doCleanFinish.func2	cleaned up work directory	{"job_id": "test"}
    t.go:81: 2022-11-24 11:37:37.091 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/runner/runner.go:194>	(*Runner).Run	sending CompletedJob	{"job_id": "test"}
    t.go:81: 2022-11-24 11:37:37.144 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/provisionerd.go:200>	(*Server).connect.func1	client stream ended
    t.go:81: 2022-11-24 11:37:37.245 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/provisionerd.go:176>	(*Server).connect	connected
    t.go:81: 2022-11-24 11:37:37.446 [INFO]	(provisionerd)	<github.com/coder/coder/provisionerd/runner/runner.go:199>	(*Runner).Run	sent CompletedJob	{"job_id": "test"}
    t.go:81: 2022-11-24 11:37:37.496 [DEBUG]	(provisionerd)	<github.com/coder/coder/provisionerd/provisionerd.go:489>	(*Server).closeWithError	closing server with error	{"error": null}

We can see that the error pops up before connected when it does, i.e. we tried to do something whilst waiting for the connection to be (re)established.

@mafredri mafredri self-assigned this Nov 24, 2022
@mafredri mafredri requested review from a team and coadler November 24, 2022 11:48
mtojek
mtojek approved these changes Nov 24, 2022
View reviewed changes
Copy link
Member

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mafredri mafredri merged commit 8ff89c4 into main Nov 24, 2022
@mafredri mafredri deleted the mafredri/flake-debug branch November 24, 2022 12:09
@github-actions github-actions bot locked and limited conversation to collaborators Nov 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@mtojek mtojek mtojek approved these changes

@coadler coadler Awaiting requested review from coadler

Assignees

@mafredri mafredri

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mafredri @mtojek