Skip to content

Handle pagination of results with more than 10,000 results gracefully #1506

Closed
@Siyfion

Description

@Siyfion

Description of the problem, including code/CLI snippet

Currently if you try to get a list with a lot of results, eg. the main GitLab project's issues list

gl = gitlab.Gitlab('https://gitlab.com', private_token='XXXXXXX')
gl.auth()
gitlab_prj = gl.projects.get(278964, lazy=True)
issues = gitlab_prj.issues.list(order_by='updated_at', sort='asc', per_page=100, as_list=False)

You'll get an error when you try and use the list:

Exception has occurred: TypeError
int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Due to the fact that, for performance reasons GitLab don't return certain headers when the responses exceed 10,000: https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Expected Behavior

Gracefully handle this lack of x-total and x-total-pages header, either by limiting the results to 10,000 responses, or allowing a user to manually set the _list._total and _list._total_pages values.

Actual Behavior

Crashes with the above exception.

Specifications

  • python-gitlab version: 2.7.1
  • API version you are using (v3/v4): v4
  • Gitlab server version (or gitlab.com): gitlab.com

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions