Skip to content

client.py doesn't retry a connection on a requests.exceptions.ConnectionError "RemoteDisconnected" #1885

Closed
@Psycojoker

Description

@Psycojoker

Description of the problem, including code/CLI snippet

On an unstable gitlab instance, sometime the instance randomly close the connection without a response, this case is not handled by python-gitlab.

Expected Behavior

Because a random connection close raise a requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) this bypass the whole retry mechanism in gitlab/client.py while using it would have solved the issue transparently.

That would likely be a try/catch to add here or something similar

result = self.session.request(
method=verb,
url=url,
json=json,
data=data,
params=params,
timeout=timeout,
verify=verify,
stream=streamed,
**opts,
)

Actual Behavior

Here is a classical traceback for this error:

Traceback (most recent call last):
  File "/home/psycojoker/.virtualenvs/cube-doctor/bin/cube-doctor", line 11, in <module>
    load_entry_point('cube-doctor', 'console_scripts', 'cube-doctor')()
  File "/home/psycojoker/code/yunohost/ynh-dev/logilab/cube-doctor/cube_doctor/doctor_hg.py", line 169, in main
    COMMANDS[args.command].workflow(
  File "/home/psycojoker/code/yunohost/ynh-dev/logilab/cube-doctor/cube_doctor/__init__.py", line 445, in wrap
    return self._workflow(*args, **kwargs)
  File "/home/psycojoker/code/yunohost/ynh-dev/logilab/cube-doctor/cube_doctor/__init__.py", line 594, in workflow
    metadata = next(modifications)
  File "/home/psycojoker/code/yunohost/ynh-dev/logilab/cube-doctor/cube_doctor/transforms/fix_deprecated_warnings.py", line 267, in modify_code
    warnings = self.get_warnings(cube)
  File "/home/psycojoker/code/yunohost/ynh-dev/logilab/cube-doctor/cube_doctor/transforms/fix_deprecated_warnings.py", line 285, in get_warnings
    for pipeline in cube.pipelines.list(
  File "/home/psycojoker/.virtualenvs/cube-doctor/lib/python3.8/site-packages/gitlab/exceptions.py", line 304, in wrapped_f
    return f(*args, **kwargs)
  File "/home/psycojoker/.virtualenvs/cube-doctor/lib/python3.8/site-packages/gitlab/mixins.py", line 242, in list
    obj = self.gitlab.http_list(path, **data)
  File "/home/psycojoker/.virtualenvs/cube-doctor/lib/python3.8/site-packages/gitlab/client.py", line 722, in http_list
    return list(GitlabList(self, url, query_data, get_next=False, **kwargs))
  File "/home/psycojoker/.virtualenvs/cube-doctor/lib/python3.8/site-packages/gitlab/client.py", line 881, in __init__
    self._query(url, query_data, **self._kwargs)
  File "/home/psycojoker/.virtualenvs/cube-doctor/lib/python3.8/site-packages/gitlab/client.py", line 891, in _query
    result = self._gl.http_request("get", url, query_data=query_data, **kwargs)
  File "/home/psycojoker/.virtualenvs/cube-doctor/lib/python3.8/site-packages/gitlab/client.py", line 594, in http_request
    result = self.session.send(prepped, timeout=timeout, **settings)
  File "/home/psycojoker/.virtualenvs/cube-doctor/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/psycojoker/.virtualenvs/cube-doctor/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

This is the related code:

cur_retries = 0
while True:
result = self.session.request(
method=verb,
url=url,
json=json,
data=data,
params=params,
timeout=timeout,
verify=verify,
stream=streamed,
**opts,
)
self._check_redirects(result)
if 200 <= result.status_code < 300:
return result
retry_transient_errors = kwargs.get(
"retry_transient_errors", self.retry_transient_errors
)
if (429 == result.status_code and obey_rate_limit) or (
result.status_code in [500, 502, 503, 504] and retry_transient_errors
):
if max_retries == -1 or cur_retries < max_retries:
wait_time = 2 ** cur_retries * 0.1
if "Retry-After" in result.headers:
wait_time = int(result.headers["Retry-After"])
cur_retries += 1
time.sleep(wait_time)
continue

Specifications

  • python-gitlab version: 2.10.1 but the related code is the same (and it appears also on the CI that pulls the latest version of python-gitlab each time)
  • API version you are using (v3/v4): v4
  • Gitlab server version (or gitlab.com): https://forge.extranet.logilab.fr/

Thanks for this lib ❤️

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions