Use separate session to download assets from AWS #288

eugef · 2014-10-29T14:38:11Z

This fixes an error "Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified" which occurs when downloading asset for the release.

The easiest fix is to use another Session as clearing headers doesn't work properly.

Steps to reproduce an issue:

Create repository, push tag 'v1' and create a release for that tag
Upload asset (for example 'build.tar.gz')
Try to use following code to download that asset

import github3

github = github3.login('username', 'password')
repository = github.repository('repository_owner', 'repository_name')

for release in repository.iter_releases():
    if release.tag_name == 'v1':
        for asset in release.assets:
            if asset.name == 'build.tar.gz':
                asset.download()

This fixes an error "Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified"

eugef · 2014-10-29T15:06:11Z

Seems, tests for Assets downloading should be modified, but this is up to you :)

eugef · 2014-10-30T08:29:17Z

@sigmavirus24 , do you have any plans to include this fix in next release? This is quite important for me as now i need to patch github3.py library and i really don't like this.

msabramo · 2014-11-01T17:15:03Z

@eugef: I just noticed that this caused a test failure under Travis CI:

___________________________ TestAsset.test_download ____________________________
self = <tests.integration.test_repos_release.TestAsset testMethod=test_download>
    def test_download(self):
        """Test the ability to download an asset."""
        cassette_name = self.cassette_name('download')
        with self.recorder.use_cassette(cassette_name,
                                        preserve_exact_body_bytes=True):
            repository = self.gh.repository('sigmavirus24', 'github3.py')
            release = repository.release(76677)
            asset = next(release.iter_assets())
            _, filename = tempfile.mkstemp()
>           asset.download(filename)
tests/integration/test_repos_release.py:70: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
github3/repos/release.py:195: in download
    if self._boolean(resp, 200, 404):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <Asset [github3.py-0.7.1-py2.py3-none-any.whl]>
response = <Response [403]>, true_code = 200, false_code = 404
    def _boolean(self, response, true_code, false_code):
        if response is not None:
            status_code = response.status_code
            if status_code == true_code:
                return True
            if status_code != false_code and status_code >= 400:
>               raise GitHubError(response)
E               github3.models.GitHubError: 403 b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Expires>2014-04-20T15:22:59Z</Expires><ServerTime>2014-10-29T15:25:17Z</ServerTime><RequestId>B8F86511682DDB91</RequestId><HostId>BATgu+moN/Z9o2dAT+gNoW8Zx3/7sX5YPTodCt71KQn+h124P1r/4GVE1S1TTdU5bFN5aT56rQc=</HostId></Error>'
github3/models.py:121: GitHubError
1 failed, 492 passed, 1 xfailed in 7.22 seconds

msabramo · 2014-11-01T17:35:43Z

The above error happens both on Travis CI and also when I checked out your branch locally and ran tox -e py27.

sigmavirus24 · 2014-11-01T17:45:02Z

The test failures are due to the fact that Betamax prefers explicit over implicit (which is contrary to how it's inspiration, VCR, works).

That said, you're trying to make a real request to the internet with an old asset id. When the tests were written, using the same session and eliminating the headers presented no issue. I need to reproduce the failure that @eugef has apparently seen before merging this (or coming up with a solution for the test problem).

msabramo · 2014-11-01T18:14:33Z

Here's how I started to look at the test failure with pdb, by using my tox.ini change in #292:

$ tox -e py27 -- --tb=short --pdb --pyargs tests.integration.test_repos_release -k test_download
GLOB sdist-make: /Users/marca/dev/git-repos/github3.py/setup.py
py27 recreate: /Users/marca/dev/git-repos/github3.py/.tox/py27
py27 installdeps: requests>=1.2.3, coverage>=3.5.2, mock>=1.0.1, pytest>=2.6.4, betamax>=0.4.1, unittest2>=0.5.1
py27 inst: /Users/marca/dev/git-repos/github3.py/.tox/dist/github3.py-0.9.2.zip
py27 runtests: PYTHONHASHSEED='1670146060'
py27 runtests: commands[0] | py.test --tb=short --pdb --pyargs tests.integration.test_repos_release -k test_download
============================================================================= test session starts ==============================================================================
platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
collected 6 items

tests/integration/test_repos_release.py F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
tests/integration/test_repos_release.py:70: in test_download
    asset.download(filename)
github3/repos/release.py:195: in download
    if self._boolean(resp, 200, 404):
github3/models.py:121: in _boolean
    raise GitHubError(response)
E   GitHubError: 403 <?xml version="1.0" encoding="UTF-8"?>
E   <Error><Code>AccessDenied</Code><Message>Request has expired</Message><Expires>2014-04-20T15:22:59Z</Expires><ServerTime>2014-11-01T18:12:07Z</ServerTime><RequestId>91D29AB28622E58B</RequestId><HostId>bkfis4scqKEuF/gBaQd97FLyVM70sPx5/Od7TdHsLM+05iGzQ8ltSUgVtHkLyGi6</HostId></Error>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/marca/dev/git-repos/github3.py/github3/models.py(121)_boolean()
-> raise GitHubError(response)
(Pdb) response
<Response [403]>
(Pdb) response.text
u'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Expires>2014-04-20T15:22:59Z</Expires><ServerTime>2014-11-01T18:12:07Z</ServerTime><RequestId>91D29AB28622E58B</RequestId><HostId>bkfis4scqKEuF/gBaQd97FLyVM70sPx5/Od7TdHsLM+05iGzQ8ltSUgVtHkLyGi6</HostId></Error>'

msabramo · 2014-11-01T18:17:36Z

Maybe betamax could steal some ideas from vcrpy?

sigmavirus24 · 2014-11-01T18:23:57Z

Ideas such as? Monkey-patching the world: no.

msabramo · 2014-11-01T18:29:47Z

I was referring to:

Betamax prefers explicit over implicit (which is contrary to how it's inspiration, VCR, works).

which admittedly, I don't understand, so I can't say anything smart. :-)

sigmavirus24 · 2014-11-01T19:35:55Z

Should have linked to the real VCR

sigmavirus24 · 2014-11-01T23:33:48Z

So here's the catch. If you do this:

import github3

r = github3.repository('sigmavirus24', 'github3.py')
release = filter(lambda _r: _r.tag_name == 'v0.7.1', r.iter_releases())[0]
asset = next(release.iter_assets())
asset.download()

It downloads perfectly for me. I even checked the MD5 checksums (downloading via asset.download() vs via the browser) and they're identical. I can not reproduce your bug @eugef

eugef · 2014-11-02T13:13:08Z

@sigmavirus24. try to download asset from the private repository - probably you will catch the bug in this case.

sigmavirus24 · 2014-11-02T14:34:28Z

I don't have a private repository to test with.

eugef · 2014-11-02T19:08:20Z

In my case i got the issue on a private repository.

sigmavirus24 · 2014-11-02T21:26:07Z

Do you have a spare private repo that I can test with?

eugef · 2014-11-03T08:52:48Z

Hi @sigmavirus24, this issue occurs only if you use github3.login() method before github3.repository() - that is why i have it with a private repo.

But you can reproduce this bug even with public repo by running this code:

import github3

g = github3.login(USERNAME, PASSWORD_OR_ACCEES_TOKEN)

r = g.repository('sigmavirus24', 'github3.py')
release = filter(lambda _r: _r.tag_name == 'v0.7.1', r.iter_releases())[0]
asset = next(release.iter_assets())
asset.download()

msabramo · 2014-11-03T14:49:17Z

Ah yes, I reproduced the failure with the example that @eugef just provided.

$ python github3_py_pr_288.py
Traceback (most recent call last):
  File "github3_py_pr_288.py", line 8, in <module>
    asset.download()
  File "/opt/doula/lib/python2.7/site-packages/github3/repos/release.py", line 194, in download
    if self._boolean(resp, 200, 404):
  File "/opt/doula/lib/python2.7/site-packages/github3/models.py", line 121, in _boolean
    raise GitHubError(response)
github3.models.GitHubError: 400 <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Basic (redacted)</ArgumentValue><RequestId>03D0215278F6A22C</RequestId><HostId>ASJkqhyGBBqMDbsouwDQEa6bpi1tpBmaPZw1x703N50tGESlANYgH2FE92+dqaNq</HostId></Error>

sigmavirus24 · 2014-11-03T16:07:03Z

In that case I know the proper way of fixing this. I'll leave this open as the bug report.

There is a great deal of complexity in requests surrounding how headers are managed and merged. Since the authentication is applied after headers are merged, using basic authentication can still be applied. Using this context manager ensures it will not be set again. Fix #288

sigmavirus24 · 2014-11-05T02:34:30Z

This has been released in 0.9.3

eugef · 2014-11-07T11:42:41Z

Thanks, it works like a charm!

sigmavirus24 · 2014-11-08T04:24:51Z

Glad to hear it @eugef and thanks for being patient

There is a great deal of complexity in requests surrounding how headers are managed and merged. Since the authentication is applied after headers are merged, using basic authentication can still be applied. Using this context manager ensures it will not be set again. Fix sigmavirus24#288

Use separate session to download assets from AWS

668f466

This fixes an error "Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified"

Use separate GitHubSession() to download assets from AWS

54f7182

eugef mentioned this pull request Oct 29, 2014

Error when downloading release assets #289

Closed

sigmavirus24 added the ready label Oct 29, 2014

sigmavirus24 added blocked Needs More Info and removed ready labels Nov 1, 2014

sigmavirus24 added in progress and removed blocked Needs More Info labels Nov 3, 2014

sigmavirus24 added the in progress label Nov 3, 2014

sigmavirus24 closed this in da20a43 Nov 5, 2014

sigmavirus24 removed the in progress label Nov 5, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use separate session to download assets from AWS #288

Use separate session to download assets from AWS #288

eugef commented Oct 29, 2014

eugef commented Oct 29, 2014

eugef commented Oct 30, 2014

msabramo commented Nov 1, 2014

msabramo commented Nov 1, 2014

sigmavirus24 commented Nov 1, 2014

msabramo commented Nov 1, 2014

msabramo commented Nov 1, 2014

sigmavirus24 commented Nov 1, 2014

msabramo commented Nov 1, 2014

sigmavirus24 commented Nov 1, 2014

sigmavirus24 commented Nov 1, 2014

eugef commented Nov 2, 2014

sigmavirus24 commented Nov 2, 2014

eugef commented Nov 2, 2014

sigmavirus24 commented Nov 2, 2014

eugef commented Nov 3, 2014

msabramo commented Nov 3, 2014

sigmavirus24 commented Nov 3, 2014

sigmavirus24 commented Nov 5, 2014

eugef commented Nov 7, 2014

sigmavirus24 commented Nov 8, 2014

Use separate session to download assets from AWS #288

Use separate session to download assets from AWS #288

Conversation

eugef commented Oct 29, 2014

eugef commented Oct 29, 2014

eugef commented Oct 30, 2014

msabramo commented Nov 1, 2014

msabramo commented Nov 1, 2014

sigmavirus24 commented Nov 1, 2014

msabramo commented Nov 1, 2014

msabramo commented Nov 1, 2014

sigmavirus24 commented Nov 1, 2014

msabramo commented Nov 1, 2014

sigmavirus24 commented Nov 1, 2014

sigmavirus24 commented Nov 1, 2014

eugef commented Nov 2, 2014

sigmavirus24 commented Nov 2, 2014

eugef commented Nov 2, 2014

sigmavirus24 commented Nov 2, 2014

eugef commented Nov 3, 2014

msabramo commented Nov 3, 2014

sigmavirus24 commented Nov 3, 2014

sigmavirus24 commented Nov 5, 2014

eugef commented Nov 7, 2014

sigmavirus24 commented Nov 8, 2014