Skip to content

Regression on calling len() on GitlabList objects since switching to keyset pagination by default #1114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Lucas-C opened this issue Jun 9, 2020 · 9 comments
Closed

Regression on calling len() on GitlabList objects since switching to keyset pagination by default #1114

Lucas-C opened this issue Jun 9, 2020 · 9 comments

Comments

@Lucas-C
Copy link

Lucas-C commented Jun 9, 2020 •
edited
Loading

Description of the problem, including code/CLI snippet

Pull request #1110 that has been released in v2.3.0 yesterday introduced a regression on a specific scenario:
when calling len() on the output of Gitlab().projects.list(as_list=False).

Expected Behavior

Being able to call len() on the iterator returned by Gitlab().projects.list(as_list=False)

Actual Behavior

Minimal reproduction code:

import gitlab
len(gitlab.Gitlab(...).projects.list(as_list=False))

Output:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    len(gitlab.Gitlab(...).projects.list(as_list=False))#, pagination='offset'))
  File ".../lib/python3.7/site-packages/gitlab/base.py", line 184, in __len__
    return len(self._list)
  File ".../lib/python3.7/site-packages/gitlab/__init__.py", line 863, in __len__
    return int(self._total)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Workaround:

import gitlab
len(gitlab.Gitlab(...).projects.list(as_list=False, pagination='offset'))

Specifications

  • python-gitlab version: sicnce v2.3.0
  • API version you are using (v3/v4): v4
  • Gitlab server version (or gitlab.com): 12.10.7-ee (80b99e43cfc)
@Lucas-C
Copy link
Author

Lucas-C commented Jun 9, 2020

#1110 introduced this code change to reflect the new pagination API of Gitlab : https://docs.gitlab.com/ce/api/README.html#keyset-based-pagination

But maybe this should not be the new value by default ?

@max-wittig
Copy link
Member

Thanks for the report. I guess this is yet another testcase for the functional tests.

Gitlab doesn't seem to include headers for keyset pagination ☹️

@max-wittig
Copy link
Member

Offset pagination doesn't work, if you have over 50.000 projects and it's recommended by GitLab.

I will look at this upstream and revert the default for right now: https://gitlab.com/gitlab-org/gitlab/-/issues/220903

@max-wittig
Copy link
Member

@Lucas-C Reverted.

@max-wittig
Copy link
Member

@Lucas-C Sadly GitLab doesn't want to continue offering the total items in keyset pagination.
See: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34273#note_361863716. I would suggest to the just follow GitLab's default pagination (with sadly is still offset) and make it easier to change to keyset (by automatically setting order_by='id' and only using keyset when in project context)

What do you think? @bufferoverflow

@Lucas-C
Copy link
Author

Lucas-C commented Jun 16, 2020 •
edited
Loading

with sadly is still offset

I think there is a small typo here, as I don't quite understand :)

I'm fine with this solution.

Overall, I'd be very happy with any solution that is backward compatible, meaning being able to call len() on .projects.list(as_list=False) output. Maybe a test for that scenario should be added by the way.

@max-wittig
Copy link
Member

I've added it already 😉 e71fe16#diff-4a8817e452ed84e4d3b53c2523dac89aR424

@Lucas-C
Copy link
Author

Lucas-C commented Jun 16, 2020

Wonderful, thanks !

@nejch nejch mentioned this issue Nov 10, 2021
Closed
@nejch
Copy link
Member

nejch commented Jan 4, 2022

This was fixed in #1773. The len() won't return anything useful (0) - but we don't have that info from the GitLab server anyway.

Feel free to reopen if it's still an issue after 3.0.0 :)

@nejch nejch closed this as completed Jan 4, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 9, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Lucas-C @max-wittig @nejch