Skip to content

Fix missing "Retry-After" header and fix integration tests #678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 13, 2019
Merged

Fix missing "Retry-After" header and fix integration tests #678

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7a3724f
fix: handle empty 'Retry-After' header from GitLab
srikanthchelluri Jan 9, 2019
16bda20
fix: remove decode() on error_message string
srikanthchelluri Jan 9, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion docs/api-usage.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,9 @@ Rate limits

python-gitlab obeys the rate limit of the GitLab server by default. On
receiving a 429 response (Too Many Requests), python-gitlab sleeps for the
amount of time in the Retry-After header that GitLab sends back.
amount of time in the Retry-After header that GitLab sends back. If GitLab
does not return a response with the Retry-After header, python-gitlab will
perform an exponential backoff.

If you don't want to wait, you can disable the rate-limiting feature, by
supplying the ``obey_rate_limit`` argument.
Expand All @@ -312,6 +314,18 @@ supplying the ``obey_rate_limit`` argument.
gl = gitlab.gitlab(url, token, api_version=4)
gl.projects.list(all=True, obey_rate_limit=False)

If you do not disable the rate-limiting feature, you can supply a custom value
for ``max_retries``; by default, this is set to 10. To retry without bound when
throttled, you can set this parameter to -1. This parameter is ignored if
``obey_rate_limit`` is set to ``False``.

.. code-block:: python

import gitlab
import requests

gl = gitlab.gitlab(url, token, api_version=4)
gl.projects.list(all=True, max_retries=12)

.. warning::

Expand Down
14 changes: 11 additions & 3 deletions gitlab/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -477,6 +477,10 @@ def http_request(self, verb, path, query_data={}, post_data=None,
# obey the rate limit by default
obey_rate_limit = kwargs.get("obey_rate_limit", True)

# set max_retries to 10 by default, disable by setting it to -1
max_retries = kwargs.get("max_retries", 10)
cur_retries = 0

while True:
result = self.session.send(prepped, timeout=timeout, **settings)

Expand All @@ -486,9 +490,13 @@ def http_request(self, verb, path, query_data={}, post_data=None,
return result

if 429 == result.status_code and obey_rate_limit:
wait_time = int(result.headers["Retry-After"])
time.sleep(wait_time)
continue
if max_retries == -1 or cur_retries < max_retries:
wait_time = 2 ** cur_retries * 0.1
if "Retry-After" in result.headers:
wait_time = int(result.headers["Retry-After"])
cur_retries += 1
time.sleep(wait_time)
continue

error_message = result.content
try:
Expand Down
2 changes: 1 addition & 1 deletion tools/python_test_v4.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -798,7 +798,7 @@
except gitlab.GitlabCreateError as e:
error_message = e.error_message
break
assert 'Retry later' in error_message.decode()
assert 'Retry later' in error_message
[current_project.delete() for current_project in projects]
settings.throttle_authenticated_api_enabled = False
settings.save()
Expand Down