Support streamed chunking (fixing #531 and others) #538
Conversation
…ing query() in chunked mode
|
i'll try to fix the test as well |
|
I think it would be cleaner to add a "stream" parameter to get a streaming response. This way we wont breakn compatibility. Perhaps there are also some usecases where someone wants to autonagically combine a chunked result in memory, like it is currently working. |
…mbines the results in ONE ResultSet, instead of many) Added documentation for stream-mode. Fixed coding style issues
|
All set and done now. There's a new stream-parameter in query() to enable "streamed chunking". This is usefull for very large result sets. (trying to traverse and anaylse over 20 million+ results here) Also updated/added documentation in the sourcecode. related project: https://github.com/psy0rz/meowton |
|
@psy0rz thank you for the patch! would you mind adding a test to this PR? |
|
Ok will do that soonish.. |
|
Hi, I believe that someone forgot to actually resolve the conflicts. @xginn8 do you mind finishing this merge PR? |
|
also i dont have time to add the requested regressions tests. |
|
If the author does not have the time to finish the PR then someone has to pick it up. I'll close this one and merge the other, if it is complete and includes tests. |
|
"Potentially break the api" ;) Dont worry you can be harsh, I would apply the same policy in your case. |
|
I had picked that up from your initial comment saying it could be backwards incompatible but this is not the case due to the steam parameter. |
|
yes, but still i could have made a mistake :) |
|
Any solution for this? returning a query with millions of rows will affect memory badly. |
|
returning a query with millions of rows is still impossible on 2019-08-01. @pedrofaria09 @aviau |
When querying large data sets, it's vital to get a chunked responses to manage memory usage. Wrapping the query response in a generator and streaming the request provides the desired result. It also fixes `InfluxDBClient.query()` behavior for chunked queries that is currently not working according to [specs](https://github.com/influxdata/influxdb-python/blob/master/influxdb/client.py#L429) Closes #585. Closes #531. Closes #538.
When querying large data sets, it's vital to get a chunked responses to manage memory usage. Wrapping the query response in a generator and streaming the request provides the desired result. It also fixes `InfluxDBClient.query()` behavior for chunked queries that is currently not working according to [specs](https://github.com/influxdata/influxdb-python/blob/master/influxdb/client.py#L429) Closes influxdata#585. Closes influxdata#531. Closes influxdata#538.
query() will now return a generator of ResultSets instead of a list when setting chunked=True. This might be backwards incompatible, but seems to be the only reason one might even use chunked mode?
Another reason to use chunked mode could be that the server enforces it, but then again: If results are that big, wouldn't you want a generator as well?