Skip to content

int() argument must be a string, a bytes-like object or a real number, not 'NoneType' while accessing pagination related attributes #1686

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
VenkatsQuest opened this issue Nov 10, 2021 · 7 comments · Fixed by #1773
Closed

int() argument must be a string, a bytes-like object or a real number, not 'NoneType' while accessing pagination related attributes #1686

VenkatsQuest opened this issue Nov 10, 2021 · 7 comments · Fixed by #1773
Assignees
@JohnVillalovos

Comments

@VenkatsQuest
Copy link

Description of the problem, including code/CLI snippet

Accessing current_page , per_page , total , total_pages attributes , throwing the following error.
int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

  gl = gitlab.Gitlab('https://gitlab.com/', private_token=developer_access_token,
                       pagination="keyset", order_by="id",
                       per_page=100)
    iterator_projects = gl.projects.list(as_list=False, owned=True)

Expected Behavior

It should return the number of pages expected

Actual Behavior

throwing exceptions.

Specifications

  • python-gitlab version:2.10.1
  • API version you are using (v3/v4): v4
  • Gitlab server version (or gitlab.com):gitlab.com
@JohnVillalovos
Copy link
Member

I can't reproduce this but I only have about 90 projects.

You might try doing:

gl.enable_debug()

As it might help diagnose the issue.

@JohnVillalovos
Copy link
Member

Also if you can paste the full Exception stack trace it would be helpful to see what part of the code is triggering the issue.

@nejch
Copy link
Member

nejch commented Nov 10, 2021

I think this is a duplicate of #1506 and maybe #1114, that gitlab.com keyset pagination thing again :(

@VenkatsQuest
Copy link
Author

Also , related to pagination i do have specific question.

Can we implement pagination in such way that , calling the required page number ( say for example 3 will return me the 3rd page ) ,

current implementation is that i need to be in the same method , iterate through page 1 , page 2 and page 3. as long as i'm in the same function all is good.

But if i leave this function , and come back to this pagination fuction with next_page number , Is this something possible ? Atleast i could not find it in the code examples.

if you guys , have some ideas , please post it here. Appreciate your support.

I'm more than willing to do code changes as well , if you point me the right souce location. Thanks much.

@kenzliang
Copy link

+1 on this issue, debug log below. Can't seem to fetch the total pages. Seems sometimes Gitlab will not return the header X-total-pages.

python-gitlab version:2.10.1
API version you are using (v3/v4): v4
Gitlab server version (or gitlab.com):13.12.12

send: b'GET /api/v4/projects/112 HTTP/1.1\r\nHost: <gitlab.private.url>\r\nUser-Agent: python-gitlab/2.10.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nPRIVATE-TOKEN: private_token\r\nContent-type: application/json\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: max-age=0, private, must-revalidate
header: Content-Encoding: gzip
header: Content-Type: application/json
header: Date: Wed, 22 Dec 2021 17:40:02 GMT
header: Etag: W/"aeeab9b1b4fe9256fb52a73786e6b849"
header: Referrer-Policy: strict-origin-when-cross-origin
header: Server: nginx
header: Strict-Transport-Security: max-age=31536000
header: Vary: Accept-Encoding
header: Vary: Origin
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-Gitlab-Feature-Category: projects
header: X-Request-Id: 01FQHKN125FFCY3RACSDX2EKPS
header: X-Runtime: 0.065188
header: Content-Length: 1398
header: Connection: keep-alive
DEBUG:urllib3.connectionpool:https://<gitlab.private.url>:443 "GET /api/v4/projects/112 HTTP/1.1" 200 1398
send: b'GET /api/v4/projects/112/repository/commits?since=2021-01-01+00%3A00%3A00&ref_name=master&per_page=100 HTTP/1.1\r\nHost: <gitlab.private.url>\r\nUser-Agent: python-gitlab/2.10.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nPRIVATE-TOKEN: private_token\r\nContent-type: application/json\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: max-age=0, private, must-revalidate
header: Content-Encoding: gzip
header: Content-Type: application/json
header: Date: Wed, 22 Dec 2021 17:40:03 GMT
header: Etag: W/"84a9393b2e899b627491c499a48ca92b"
header: Link: <https://<gitlab.private.url>/api/v4/projects/112/repository/commits?id=112&order=default&page=2&per_page=100&ref_name=master&since=2021-01-01T00%3A00%3A00%2B00%3A00>; rel="next", <https://<gitlab.private.url>/api/v4/projects/112/repository/commits?id=112&order=default&page=1&per_page=100&ref_name=master&since=2021-01-01T00%3A00%3A00%2B00%3A00>; rel="first", <https://<gitlab.private.url>/api/v4/projects/112/repository/commits?id=112&order=default&page=2&per_page=100&ref_name=master&since=2021-01-01T00%3A00%3A00%2B00%3A00>; rel="last"
header: Referrer-Policy: strict-origin-when-cross-origin
header: Server: nginx
header: Strict-Transport-Security: max-age=31536000
header: Vary: Accept-Encoding
header: Vary: Origin
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-Gitlab-Feature-Category: source_code_management
header: X-Next-Page: 2
header: X-Page: 1
header: X-Per-Page: 100
header: X-Prev-Page: 
header: X-Request-Id: 01FQHKN17NPZMT9XGVAHH1QAXX
header: X-Runtime: 0.065319
header: Content-Length: 12033
header: Connection: keep-alive
DEBUG:urllib3.connectionpool:https://<gitlab.private.url>:443 "GET /api/v4/projects/112/repository/commits?since=2021-01-01+00%3A00%3A00&ref_name=master&per_page=100 HTTP/1.1" 200 12033
Traceback (most recent call last):
  File "python_gitlab.py", line 83, in <module>
    get_repo_merges(SESSION, r_id,r_name)
  File "python_gitlab.py", line 32, in get_repo_merges
    print(commits.total)
  File "python/lib/python3.9/site-packages/gitlab/base.py", line 259, in total
    return self._list.total
  File "python/lib/python3.9/site-packages/gitlab/client.py", line 963, in total
    return int(self._total)

@kenzliang
Copy link

seems to be the issue here:
https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

JohnVillalovos added a commit that referenced this issue Dec 22, 2021
@JohnVillalovos
fix: handle situation where gitlab.com does not return values
c79e061 
If a query returns more than 10,000 records than the following values
are NOT returned:
  x.total_pages
  x.total

Modify the code to allow no value to be set for these values. If there
is not a value returned the functions will now return None.

https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Closes #1686
@JohnVillalovos JohnVillalovos mentioned this issue Dec 22, 2021
Merged
@JohnVillalovos
Copy link
Member

+1 on this issue, debug log below. Can't seem to fetch the total pages. Seems sometimes Gitlab will not return the header X-total-pages.

Thank you very much for the debug log and the link. That made it easy to figure out the issue. I have proposed a PR to resolve the issue.

@JohnVillalovos JohnVillalovos self-assigned this Dec 22, 2021
JohnVillalovos added a commit that referenced this issue Dec 22, 2021
@JohnVillalovos
fix: handle situation where GitLab does not return values
ca2e8e0 
If a query returns more than 10,000 records than the following values
are NOT returned:
  x.total_pages
  x.total

Modify the code to allow no value to be set for these values. If there
is not a value returned the functions will now return None.

https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Closes #1686
JohnVillalovos added a commit that referenced this issue Dec 22, 2021
@JohnVillalovos
fix: handle situation where GitLab does not return values
971a617 
If a query returns more than 10,000 records than the following values
are NOT returned:
  x.total_pages
  x.total

Modify the code to allow no value to be set for these values. If there
is not a value returned the functions will now return None.

https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Closes #1686
JohnVillalovos added a commit that referenced this issue Dec 25, 2021
@JohnVillalovos
fix: handle situation where GitLab does not return values
4690dc2 
If a query returns more than 10,000 records than the following values
are NOT returned:
  x.total_pages
  x.total

Modify the code to allow no value to be set for these values. If there
is not a value returned the functions will now return None.

https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Closes #1686
JohnVillalovos added a commit that referenced this issue Dec 30, 2021
@JohnVillalovos
fix: handle situation where GitLab does not return values
4a17937 
If a query returns more than 10,000 records than the following values
are NOT returned:
  x.total_pages
  x.total

Modify the code to allow no value to be set for these values. If there
is not a value returned the functions will now return None.

https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Closes #1686
JohnVillalovos added a commit that referenced this issue Dec 30, 2021
@JohnVillalovos
fix: handle situation where GitLab does not return values
c308c30 
If a query returns more than 10,000 records than the following values
are NOT returned:
  x.total_pages
  x.total

Modify the code to allow no value to be set for these values. If there
is not a value returned the functions will now return None.

Update unit test so no longer `xfail`

https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Closes #1686
JohnVillalovos added a commit that referenced this issue Dec 30, 2021
@JohnVillalovos
fix: handle situation where GitLab does not return values
4947901 
If a query returns more than 10,000 records than the following values
are NOT returned:
  x.total_pages
  x.total

Modify the code to allow no value to be set for these values. If there
is not a value returned the functions will now return None.

Update unit test so no longer `xfail`

https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Closes #1686
JohnVillalovos added a commit that referenced this issue Dec 30, 2021
@JohnVillalovos
fix: handle situation where GitLab does not return values
cb824a4 
If a query returns more than 10,000 records than the following values
are NOT returned:
  x.total_pages
  x.total

Modify the code to allow no value to be set for these values. If there
is not a value returned the functions will now return None.

Update unit test so no longer `xfail`

https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers

Closes #1686
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@JohnVillalovos JohnVillalovos

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: handle situation where gitlab.com does not return values python-gitlab/python-gitlab
4 participants
@JohnVillalovos @nejch @VenkatsQuest @kenzliang