Skip to content
This repository was archived by the owner on Oct 29, 2024. It is now read-only.

chunked parameter in query method doesn't work #21

Closed
pkaeding opened this issue Apr 29, 2014 · 10 comments
Closed

chunked parameter in query method doesn't work #21

pkaeding opened this issue Apr 29, 2014 · 10 comments
Assignees
Labels

Comments

@pkaeding
Copy link

It seems that this library will just try to parse as json the full string, once it all comes down, which:

  • defeats the purpose of streaming the chunks
  • won't work, since there will be many JSON objects in the stream

I'm not sure that this is really possible using requests without jumping through some crazy hoops.

@ilovenwd
Copy link

@manugarri
Copy link
Contributor

hmm any update on this? I am having issues querying large amounts of points. I was considering writing a batch parameter for the client.query() method, but id rather use something that is already implemented.

@gourneau
Copy link

This is still a problem. :(

@gourneau
Copy link

Here is a patch that will make chucked work with 1 series of data. I am sure there are plenty of cases where it will not work though.

#45

@aviau aviau added the bug label Nov 3, 2014
@aviau
Copy link
Collaborator

aviau commented Nov 3, 2014

I'll be taking a look at this soon.

@aviau aviau self-assigned this Nov 3, 2014
aviau pushed a commit that referenced this issue Nov 5, 2014
@aviau
Copy link
Collaborator

aviau commented Nov 5, 2014

Now that we use python-requests's .json() method. It does not seem to be an issue. If I am wrong, please provide me with an example of a response that does not work.

Thank you for reporting the issue :)

@aviau
Copy link
Collaborator

aviau commented Nov 5, 2014

I have realised that my test case only checks for a response with one object in it. I'll re-open this.

@aviau aviau reopened this Nov 5, 2014
@gourneau
Copy link

gourneau commented Nov 5, 2014

Thanks @ReAzem !

@aviau
Copy link
Collaborator

aviau commented Nov 6, 2014

@gourneau, @pkaeding, @ilovenwd , @manugarri

I have pushed a fix proposition to branch fix_21_proposition (1a523a4).

What do you guys think?

  • It does not add any other library
  • It's safer than altering the string (@gourneau's proposition), but surely not faster. Is speed even a concern?

I have not found any better ways of doing this with the standard lib, but this solutions works pretty well, I think.

@aviau aviau closed this as completed in 1a523a4 Nov 11, 2014
@cbarraford
Copy link

I opened a PR that is relating to this ticket. When pulling down chunked response its nice for the server side as it utilizes less memory, but client side you'd have to store it all in memory which wouldn't work if you're querying for GBs of data. This PR allows you to stream directly to disk.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

6 participants