Skip to content
This repository was archived by the owner on Oct 29, 2024. It is now read-only.

gzip support on influxdb-python #718

Open
philip-jones opened this issue May 26, 2019 · 10 comments
Open

gzip support on influxdb-python #718

philip-jones opened this issue May 26, 2019 · 10 comments

Comments

@philip-jones
Copy link

Hi,

Great API, we use it heavily. Thank you.

The equivalent Java API influxdb-java, which I also use, allows gzip compression to be explicitly enabled when setting up a connection, e.g.

final javaInfluxClient = InfluxDBFactory.connect(SERVER, USERNAME, PASSWORD);
javaInfluxClient.enableGzip();
// ...etc

Perhaps this is enabled by default in influxdb-python, however I cannot find any reference to gzip compression. Perhaps I am looking in the wrong place.

This would be very useful to allow a reduction in bandwidth for IoT devices with very small data tariffs.

Would it be possible to add this? I'm happy to have a go at writing a patch for this but would like to get some feedback from the maintainers first.

Thank you, Phil.

@driftregion
Copy link

Related: #451
In Oct 2017, xginn8 mentioned that there was no InfluxDB API documentation for gzip compression. Perhaps this has changed since then?
https://docs.influxdata.com/influxdb/v1.7/guides/writing_data/#configure-gzip-compression

@KEClaytor
Copy link
Contributor

Hello, I just wanted to jump in and let you know that I've been able to add gzip compression to the client and have made a PR #732

I agree that data compression is really important, especially on IoT devices and mobile networks.

I'm unsure how to properly do the tests so it may still be awhile until it's merged.

@philip-jones
Copy link
Author

Thank you for the response KEClaytor & driftregion, much appreciated.

Will be happy to do some testing when you are ready / there is a consensus about how this should be done.

@lovasoa
Copy link
Contributor

lovasoa commented Jul 11, 2019

Related: #733.
MessagePack greatly reduces network traffic between the client and the server.

@KEClaytor
Copy link
Contributor

@driftregion Yes, compression seems to be supported. Thanks for linking to the official docs on this. The earliest mention of it in the changelog is in v1.0:
https://docs.influxdata.com/influxdb/v1.7/about_the_project/releasenotes-changelog/

@lovasoa Message Pack looks really neat. The only reference I can find to it in the docs is:

  • Add message pack format for query responses.

From: https://docs.influxdata.com/influxdb/v1.7/about_the_project/releasenotes-changelog/

Do you know if they also support message pack for transmitting data? If it's just for the query responses then gzip'd data for transmission still makes sense. I'd prefer to see some mention of it on their official docs page, but we could probably implement it the same way I did gzip - just compress / convert the message before write.

Since gzip seems to be supported back to v1.0 and for transmission and receiving that would be my preferred way to go.

@philip-jones
Copy link
Author

It's really great that you are all jumping in on this request, thank you.

As I (inappropriately) posted on the related MessagePack issue, I think MessagePack is a great alternative to JSON, but I also suspect that it is orthogonal to the issue of using gzip for HTTPS data transmission?

I suspect both JSON and MessagePack containing perhaps hundreds of data points, probably with highly repetitive field names, would be served very well by gzip compression? On that basis I would avoid conflating the issues.

Happy to be corrected regarding this if I've got the wrong end of the stick.

cheers, Phil.

@lovasoa
Copy link
Contributor

lovasoa commented Jul 21, 2019

@KEClaytor No, msgpack doesn't seem to be supported for sending data. See the corresponding pull request.

@philip-jones
Copy link
Author

Hi all,

I was wondering if there was any chance of moving this request forwards? (Specifically, implementing / rolling out gzip compression).

IoT clients still crying out for some compression :-)

kind regards & thank you,

Phil.

@malcolmwhat
Copy link

malcolmwhat commented Oct 2, 2020

Oh I think this is done (I'm pretty sure this issue can be closed), I checked this issue because I was getting a weird key error related to the constructor when providing the gzip argument...

The InfluxDBClient documentation lists gzip as an argument to the constructor as a keyword argument. Instead of setting it after the constructor like you refer to in the Java example, you set it right in the constructor of the client like this:

client = InfluxDBClient(host, port, u, p, 'example', gzip=True)   # By default in the doc, gzip=False

@chriszal
Copy link

@malcolmwhat After you enabled gzip on the constructor how exactly did you manage to write a gzipped file with the library?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants