GitLab Support Submit a Ticket Dacoda Strack

GitLab, Inc. > My activities

Hanging git operations over HTTP

(but not SSH)
Dacoda Strack Requester Dacoda Strack

1 year ago
Created February 13, 2023
12:59
We rebooted our standalone Gitaly EC2 instance yesterday after a disk space
issue. Last activity March 12, 2024 17:11

CCs Paul Duffy

Ever since that reboot, git operations over HTTP have been hanging forever.
Matthew
Vanderhoof
We are running gitlabsos on all 4 nodes—one Gitaly node, and three application
Brad Downey
server nodes with the following services Rick Landon

• gitlab-exporter
• gitlab-pages Assigned to Diana Stanley

• gitlab-workhorse
Id #374094
• logrotate
• nginx Status Solved

• node-exporter
Tell us about Premium customer
• puma your GitLab
• registry subscription

• sidekiq
 Problem type Problems, errors, and
performance issues
Dacoda Strack
with GitLab
1 year ago
components

Attaching GitLab SOS output, and will by attaching the various `gitlab.rb` files What area(s) Gitaly, Nginx,
soon. Architecture diagram... forth...coming? within the Other/Unsure, Puma
GitLab
redacted.txt components is
this
1 Byte · Download
concerning?
redacted.txt
What install sm_install_type_aws_quickstart
1 Byte · Download
method do you
redacted.txt use?

1 Byte · Download
GitLab Version 14.7.7
redacted.txt
What is the Severity 2 - GitLab is
1 Byte · Download
ticket severity? highly degraded

Priority High
Dacoda Strack
1 year ago Preferred Americas, USA
region for
Instead of an Architecture diagram, how about a CloudFormation stack tem- support
plate? Let us know if this is not acceptable

Attachments

redacted.txt
1 Byte · Download

Dacoda Strack
1 year ago

Here is that CloudFormation stack template. By the way, my apologies, I have

put the indicated the installation method as "AWS QuickStart for GL Cloud
Native Hybrid", but in fact I think we may have stood up our AWS deployment
redacted.txt
before this was an official AWS partner solution and so our architecture might 1 Byte · Download
be whack and different redacted.txt
1 Byte · Download
My apologies redacted.txt
1 Byte · Download
redacted.txt
redacted.txt
1 Byte · Download 1 Byte · Download

redacted.txt
Dacoda Strack 1 Byte · Download

1 year ago redacted.txt

1 Byte · Download
Also unrelated, but I would like to extend my deepest sympathies to the large
layoff that just hit you guys and I hope that morale has not been too incredibly
affected. Love y'all

Thanks

Dacoda Strack
1 year ago

I am a fool for not attempting to bounce the application server nodes, or at least
restarting the gitlab-workhorse process. This appears to have fixed the issue for
us, but I'm surprised that the gitlab-workhorse logs were not helpful at all in this
regard

Dacoda Strack
1 year ago

I'm very curious why the gitlab-workhorse process needed to have been
restarted. Is there any reference to documentation that indicates to restart the
gitlab-workhorse process on the application server nodes when the Gitaly
process host is rebooted or when the Gitaly process is otherwise disturbed?

Does the gitlab-workhorse hold on to a connection pool that is not

healthchecked, and so it keeps trying dead TCP connections without making
new ones? Where are these logs?

I am a fool, but I am also surprised

James Reed
1 year ago

Hi Dacoda,

I see that an emergency ticket #374146 was raised and the following response
sent out by Diana S on our end:

I’m going to close out this ticket and follow up in your other ticket. Thanks
for uploading GitLabSOS files. I’m going to take a look at those and get
back with you as soon as I can.

Thanks,

Diana

As I understand it, you've since resolved the issue by restarting the gitlab-
workhorse process, and at this point in time you're after a bit more detail on
what may have happened here/why, per your recent questions. If for some rea-
son this is incorrect and you do actually still have problems with git opera-
tions over HTTP(s) , please let us know in your response. In any case, we'll re-
view your GitLabSOS logs when we can, and then proceed from there.
Regards,

James Reed
Support Engineer
GitLab Support

Want to submit feedback on this ticket so far? Click here to tell us how we're do-
ing!

Dacoda Strack
1 year ago

As I understand it, you've since resolved the issue by restarting the

gitlab-workhorse process, and at this point in time you're after a bit
more detail on what may have happened here/why, per your recent ques-
tions. If for some reason this is incorrect and you do actually still have prob-
lems with git operations over HTTP(s) , please let us know in your re-
sponse.

That is correct, we've resolved the issue by restart the gitlab-workhorse

process, but we'd like a bit more information for why this happened and how we
could have avoided it/avoided so long a degraded GitLab.

There are no longer problems with git operations over HTTP(s)

Diana Stanley
1 year ago

Request #374146 "Re: Need help escalating Gitlab ..." was closed and merged
into this request. Last comment in request #374146:
Thank you very much,
Dacoda Strack

From: Diana Stanley (GitLab Support) emergency@gitlab.com

Sent: Monday, February 13, 2023 2:23 PM
To: Dacoda Strack Dacoda.Strack@panasonic.aero
Subject: [GitLab Support] Re: Re: Need help escalating Gitlab 504 clone issue -
ticket (374094)

EXTERNAL EMAIL: This email originated from a sender outside of the organi-
zation. Use caution when responding to requests, clicking links, or opening at-
tachments.

Diana Stanley
1 year ago

Dacoda,

I'm not certain exactly what might have gone wrong here, but I do see a lot of
these errors in the gitaly log:

{"level":"warning","msg":"[core] grpc: Server.Serve failed to create ServerTransp

{"level":"warning","msg":"[core] grpc: Server.Serve failed to create ServerTranspo
{"level":"warning","msg":"[core] grpc: Server.Serve failed to create ServerTranspo

And similarly I see a lot of these in the workhorse logs:

{"correlation_id":"01GS62KQZ2BKN3B6ACW2PDFS06","error":"handleGetInfoRefs: rpc er
{"correlation_id":"01GS62M7VMQ02WP8C4X53MX929","error":"handleGetInfoRefs: rpc err
{"correlation_id":"01GS62PK36CVP1SFBNEVCZGJEB","error":"handleGetInfoRefs: rpc err

I think these errors on both sides are symptoms of whatever the problem was,
but I can't pinpoint the exact nature of it with these errors. We do periodically
see rare cases where we have to manually restart processes, usually associ-
ated with system upgrades. I'm not aware of any particular issue around restart-
ing gitaly that would be related to this.

Thanks,
Diana

Diana Stanley
Senior Support Engineer
GitLab Support

Want to submit feedback on this ticket so far? Click here to tell us how we're do-
ing!

Dacoda Strack
1 year ago

I feel as if those "context canceled" errors and perhaps the EOF Gitaly errors as
well may have been from the AWS ALB timing out at 60 seconds

I am curious if... I am curious if we had performed a large number of requests

directly hitting the GitLab application server (circumventing the load balancer) if
we would have a better sample with which to explore the issue

As it stands, we only issued a handful of requests directly to one of the applica-

tion server nodes, and I'm unsure if I kept the correlation IDs around.

I wonder if I can attempt to reproduce the environment in our development in-

stance by rebooting Gitaly while keeping the application nodes up and running.

I may try briefly tomorrow, but will probably not spend more than... say an hour
or so tomorrow attempting to recreate.

Asmaa Hassan Ahmed Ali

1 year ago

Hi Dacoda,

That might help indeed. I'll mark the ticket as pending while we wait for your
feedback.

Regards,

Asmaa Hassan Ahmed Ali

Senior Support Engineer
GitLab Support

Want to submit feedback on this ticket so far? Click here to tell us how we're do-
ing!

GitLab SupportOps Bot

1 year ago

Hello,

This is a follow up for your ticket (#374094) with GitLab. We haven't heard from
you for a few days. Perhaps you missed our last message?

Please reply if you'd still like help or want us to close the ticket. We'll be happy
to assist.

GitLab SupportOps Bot

 1 year ago

Hello,

This is a 2nd follow up for your ticket (#374094) with GitLab. Perhaps you
missed our previous messages?

To make sure everything's OK, we'll send an email asking for your feedback to-
morrow.

Please reply if you'd still like help and we'll be happy to assist.

This request is closed for comments. You can create a follow-up.

GitLab Support
GitLab's Terms of use
GitLab's Privacy policy