Skip to content

projects.all() returns only 20 projects #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
godmar opened this issue Feb 10, 2016 · 6 comments
Closed

projects.all() returns only 20 projects #93

godmar opened this issue Feb 10, 2016 · 6 comments

Comments

@godmar
Copy link

godmar commented Feb 10, 2016

In issue #53 I had asked how to list all projects. The answer was pagination.
I am now trying the same with the current version of python-gitlab. This time, I am using gl.projects.all() as recommended in the deprecation warning I received with my old code.

However, it seems the code isn't doing any pagination (is it no longer necessary? In Issue #53, you said there's no way to avoid it?

As a result, I'm only getting 20 projects returned.

What's the way to iterate over all projects using python-gitlab?

@gpocentek
Copy link
Contributor

There's some inconsistency in the listing methods. The managers' list()method can return all the items with all=True, so this might do what you're looking for:

all_projects = gl.projects.list(all=True)

This is not supported on other listing method (all, search, owned) for the moment.

@godmar
Copy link
Author

godmar commented Feb 11, 2016

gl.projects.list(all=True) returns 10 projects only. gl.projects.all() returns 20 projects.

If you look at the code: projects.all() calls custom_list, which makes a single API request.

projects.list call gl.list(), which has some code here but that stops after 10 results because there's no 'next' field in the response. The returned response is:

{'last': {'url': 'https://git.cs.vt.edu/api/v3/projects?page=1&per_page=0', 'rel': 'last'}, 'first': {'url':
'https://git.cs.vt.edu/api/v3/projects?page=1&per_page=0', 'rel': 'first'}}

which does not have 'next' property.

How do I iterate to return all projects in the new code base?

@godmar
Copy link
Author

godmar commented Feb 11, 2016

ps: gl.projects.all() receives this API response:

{'first': {'url': 'https://git.cs.vt.edu/api/v3/projects/all?page=1&per_page=0', 'rel': 'first'}, 'last': {'url': 'https://git.cs.vt.edu/api/v3/projects/all?page=38&per_page=0', 'rel': 'last'}, 'next': {'url': 'https://git.cs.vt.edu/api/v3/projects/all?page=2&per_page=0', 'rel': 'next'}}

but then fails to process the 'next' field as far as I can see.

How is this supposed to work?

@godmar
Copy link
Author

godmar commented Feb 11, 2016

Oh, and I know why. projects.all() issues a request for /projects/all, but then doesn't process the 'next' field in the r.links. projects.list(all=True) issues a request for /projects, in which case the API response doesn't have a 'next' field.

@gpocentek
Copy link
Contributor

OK, I think I get what you need. You can still iterate using the page and per_page arguments.

l = []; page = 0
while True:
  next_list = gl.projects.all(per_page=50, page=page)
  if not next_list:
    break
  l.extend(next_list)
  page += 1

I'll implement the all argument support in _custom_list to ease this as soon as I can.

Thanks.

@gdubicki
Copy link

gdubicki commented Apr 17, 2016

I just run into this limitation and had to use code from #93 (comment) as a workaround. So in my opinion this issue should not be closed. Sorry, I mixed things up. Actually gl.projects.all(all=True) worked ..probably as my Gitlab timeouted on this request (we have A LOT of projects) and changing it's settings didn't seem to help. So the above code was a workaround for my timeouting issue.

jkogut added a commit to jkogut/python-gitlab that referenced this issue May 28, 2019
Having exactly 20 internal and 5 private projects in the group
spent some time debugging this issue.

Hopefully that helped: python-gitlab#93

Imho should be definitely mention about `all=True` parameter.
max-wittig pushed a commit that referenced this issue Jul 21, 2019
Having exactly 20 internal and 5 private projects in the group
spent some time debugging this issue.

Hopefully that helped: #93

Imho should be definitely mention about `all=True` parameter.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 29, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants