Skip to content
This repository was archived by the owner on Oct 29, 2024. It is now read-only.

Add support for streamed chunks when making large queries #478

Closed
msohcw opened this issue Jul 18, 2017 · 3 comments
Closed

Add support for streamed chunks when making large queries #478

msohcw opened this issue Jul 18, 2017 · 3 comments

Comments

@msohcw
Copy link

msohcw commented Jul 18, 2017

Currently I've written a system that takes advantage to requests streaming capability to avoid loading all of the chunks into memory at once. Generally this seems faster and better than what is currently provided which loads all the chunks immediately. Could support be added for a generator that lazily loads chunks from buffer?
@mpas
Copy link

mpas commented Jan 4, 2018

I have a need to export a large number of rows and seem to run in the same issue that you have solved. Could your solution be used to export these large numbers?

@msohcw
Copy link
Author

msohcw commented Jan 5, 2018 •
edited
Loading

Essentially, I wrote a wrapper around InfluxDB and extended it with a lazy_query method. I then used requests in-built streaming capability [1] to stream HTTP requests to InfluxDB. I would then return the stream as a generator.

[1] http://docs.python-requests.org/en/master/user/advanced/

@sebito91
Copy link
Contributor

Fixed in #753

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mpas @msohcw @sebito91