feat: Add play command to project pipeline schedules #1069

twonds · 2020-04-09T00:04:35Z

I have added a custom play command to the project pipeline schedules.

I made a previous pull request #1068 but the commits were incorrect. I also fixed the lint errors.

Please let me know if I am missing anything else.

twonds · 2020-04-09T00:05:52Z

Where would I add tests for this?

max-wittig

Thanks for your MR! I've added a few comments

gitlab/v4/objects.py

gitlab/__init__.py

max-wittig · 2020-04-09T07:43:35Z

Tests are a bit of a mess right now. You can add your tests here: https://github.com/python-gitlab/python-gitlab/blob/master/gitlab/tests/test_gitlab.py or use the newer structure and create a new file here
https://github.com/python-gitlab/python-gitlab/tree/master/gitlab/tests/objects

twonds · 2020-04-09T18:30:15Z

Thanks for the feedback! I will make these changes!

twonds · 2020-04-13T22:54:52Z

I am having a tough time getting the tests to execute. Which command should I use? The python functionals tests for v4?

When I run tox -e py_func_v4 2>&1|tee test.log Things seem ok but the configuration isn't valid.

I get the following error.

gitlab Reconfigured!
Config file content (/tmp/python-gitlab.cfg):
[global]
default = local
timeout = 30

[local]
url = http://localhost:8080
private_token = 
api_version = 4
Test environment initialized.
Traceback (most recent call last):
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/exceptions.py", line 279, in wrapped_f
    return f(*args, **kwargs)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/mixins.py", line 470, in download
    path, streamed=streamed, raw=True, **kwargs
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/__init__.py", line 599, in http_get
    "get", path, query_data=query_data, streamed=streamed, **kwargs
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/__init__.py", line 574, in http_request
    response_body=result.content,
gitlab.exceptions.GitlabHttpError: 404: 404 Not found or has expired

max-wittig · 2020-04-14T14:43:08Z

You can use tox.

tox -e py_func_v4 # for functional tests (will spawn a docker container...)
tox -e py38 # for unit tests

max-wittig · 2020-04-14T14:45:52Z

I've updated the test docs: #1073

nejch · 2020-04-16T08:06:39Z

I am having a tough time getting the tests to execute. Which command should I use? The python functionals tests for v4?

When I run tox -e py_func_v4 2>&1|tee test.log Things seem ok but the configuration isn't valid.

I get the following error.

gitlab Reconfigured!
Config file content (/tmp/python-gitlab.cfg):
[global]
default = local
timeout = 30

[local]
url = http://localhost:8080
private_token = 
api_version = 4
Test environment initialized.
Traceback (most recent call last):
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/exceptions.py", line 279, in wrapped_f
    return f(*args, **kwargs)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/mixins.py", line 470, in download
    path, streamed=streamed, raw=True, **kwargs
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/__init__.py", line 599, in http_get
    "get", path, query_data=query_data, streamed=streamed, **kwargs
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/__init__.py", line 574, in http_request
    response_body=result.content,
gitlab.exceptions.GitlabHttpError: 404: 404 Not found or has expired

@twonds the command as Max said above should execute the tests. In your example above, did you remove the private_token from the output or is it empty? That would be an issue.

But the (rather cryptic) 404 error suggests something's wrong with the Project or Group Export test (see 404 Not found or has expired). This is unrelated to your changes - the Group Export API is still a bit incomplete so maybe we're not waiting long enough for export creation. Do you consistently get this same error if you re-run tests?

twonds · 2020-04-20T17:16:00Z

Thanks! Unit tests work and trying functional tests now. I will implement a test and update this pr

twonds · 2020-04-20T17:24:47Z

This time on the functional tests gitlab seemed to come up but the tests did not execute due to a timeout. This timeout could be anything, I have not investigated. What are the minimum resources for running gitlab locally?

Waiting for GitLab to reconfigure.. (130s)
Waiting for GitLab to reconfigure.. (135s)
gitlab Reconfigured!
Config file content (/tmp/python-gitlab.cfg):
[global]
default = local
timeout = 30

[local]
url = http://localhost:8080
private_token = 8yVpYsVj_LQWx_D-EANr
api_version = 4
Test environment initialized.
	Traceback (most recent call last):
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=8080): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/tools/python_test_v4.py", line 526, in <module>
    discussion = commit.discussions.get(discussion.id)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/exceptions.py", line 279, in wrapped_f
    return f(*args, **kwargs)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/mixins.py", line 50, in get
    server_data = self.gitlab.http_get(path, **kwargs)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/__init__.py", line 599, in http_get
    "get", path, query_data=query_data, streamed=streamed, **kwargs
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/gitlab/__init__.py", line 537, in http_request
    result = self.session.send(prepped, timeout=timeout, **settings)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/christopherzo/src/github.com/zillow/python-gitlab/.tox/py_func_v4/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8080): Read timed out. (read timeout=30)
ERROR: 'python /Users/christopherzo/src/github.com/zillow/python-gitlab/tools/python_test_v4.py' failed
Deactivating Python virtualenv...
Deleting python virtualenv...
Stopping gitlab-test docker container...
Done.
ERROR: InvocationError for command /Users/christopherzo/src/github.com/zillow/python-gitlab/tools/py_functional_tests.sh -a 4 (exited with code 1)
_________________________________________________________________________________________________ summary _________________________________________________________________________________________________
ERROR:   py_func_v4: commands failed

nejch · 2020-04-20T18:47:12Z

The Travis builds run on 2 cores with 7.5GB, but the functional tests do fail sometimes. I've also had issues with an 8GB VM with 2 cores.

It's actually possible the tests were running but failed somewhere randomly on timeout. Currently there's still not much visibility into what's happening with functional tests unless they explicitly fail on an assert. Is this consistently failing if you retry?

fix: remove version from setup feat: add pipeline schedule play error exception docs: add documentation for pipeline schedule play

twonds · 2020-04-20T23:19:23Z

Is this consistently failing if you retry?

Yes, same error for each attempt

twonds · 2020-04-20T23:20:18Z

I have added a test

nejch · 2020-04-21T05:41:24Z

@twonds I think maybe your test was lost in your last force-push 😊

twonds · 2020-04-21T16:28:49Z

I think maybe your test was lost in your last force-push 😊

Hah, I have been a poor contributor in this pull request. I can not get used to commit lint, causing git rebase screw ups. I also can not get functional tests working. :)

nejch · 2020-04-21T22:43:02Z

I think maybe your test was lost in your last force-push blush

Hah, I have been a poor contributor in this pull request. I can not get used to commit lint, causing git rebase screw ups. I also can not get functional tests working. :)

Not at all, you've been very responsive :)

I can't see what you've tried for functional tests, but I just had a look and realized the functional tests would take a bit of scaffolding before you could even add a schedule play test, we'd first need a project with pipelines created and schedules etc, which aren't yet tested atm. So I wouldn't worry about it if it's too convoluted for a first contribution.

Since you have unit tests now, I'd just get the pipeline green again (run black with tox -e black, it's a bit opinionated about style so might clash with your linter) and ping the reviewers again ;)

twonds · 2020-04-22T00:07:43Z

Thanks @nejch . @max-wittig I think this is ready. Thanks for your patience!

max-wittig

@twonds Thanks for your patience!

twonds force-pushed the feat/add-custom-pipeline-schedule-play branch from fc76496 to 48bc54f Compare April 9, 2020 00:19

max-wittig requested changes Apr 9, 2020

View reviewed changes

gitlab/v4/objects.py Outdated Show resolved Hide resolved

gitlab/v4/objects.py Outdated Show resolved Hide resolved

gitlab/v4/objects.py Outdated Show resolved Hide resolved

gitlab/v4/objects.py Outdated Show resolved Hide resolved

gitlab/__init__.py Outdated Show resolved Hide resolved

max-wittig assigned twonds Apr 9, 2020

twonds requested a review from max-wittig April 13, 2020 22:52

feat: add play command to project pipeline schedules

07b9988

fix: remove version from setup feat: add pipeline schedule play error exception docs: add documentation for pipeline schedule play

twonds force-pushed the feat/add-custom-pipeline-schedule-play branch from 212ea77 to 07b9988 Compare April 20, 2020 23:18

ci: add a test for creating and triggering pipeline schedule

9f04560

ci: lint fixes

930122b

max-wittig approved these changes Apr 22, 2020

View reviewed changes

max-wittig merged commit 9d66cb3 into python-gitlab:master Apr 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add play command to project pipeline schedules #1069

feat: Add play command to project pipeline schedules #1069

twonds commented Apr 9, 2020 •

edited

Loading

twonds commented Apr 9, 2020

max-wittig left a comment

max-wittig commented Apr 9, 2020

twonds commented Apr 9, 2020

twonds commented Apr 13, 2020

max-wittig commented Apr 14, 2020

max-wittig commented Apr 14, 2020

nejch commented Apr 16, 2020

twonds commented Apr 20, 2020

twonds commented Apr 20, 2020

nejch commented Apr 20, 2020

twonds commented Apr 20, 2020

twonds commented Apr 20, 2020

nejch commented Apr 21, 2020

twonds commented Apr 21, 2020 •

edited

Loading

nejch commented Apr 21, 2020

twonds commented Apr 22, 2020

max-wittig left a comment

feat: Add play command to project pipeline schedules #1069

feat: Add play command to project pipeline schedules #1069

Conversation

twonds commented Apr 9, 2020 • edited Loading

twonds commented Apr 9, 2020

max-wittig left a comment

Choose a reason for hiding this comment

max-wittig commented Apr 9, 2020

twonds commented Apr 9, 2020

twonds commented Apr 13, 2020

max-wittig commented Apr 14, 2020

max-wittig commented Apr 14, 2020

nejch commented Apr 16, 2020

twonds commented Apr 20, 2020

twonds commented Apr 20, 2020

nejch commented Apr 20, 2020

twonds commented Apr 20, 2020

twonds commented Apr 20, 2020

nejch commented Apr 21, 2020

twonds commented Apr 21, 2020 • edited Loading

nejch commented Apr 21, 2020

twonds commented Apr 22, 2020

max-wittig left a comment

Choose a reason for hiding this comment

twonds commented Apr 9, 2020 •

edited

Loading

twonds commented Apr 21, 2020 •

edited

Loading