Skip to content
This repository was archived by the owner on Oct 29, 2024. It is now read-only.

Properly use chunk_size param #524

Closed
anthonyserious opened this issue Oct 25, 2017 · 2 comments
Closed

Properly use chunk_size param #524

anthonyserious opened this issue Oct 25, 2017 · 2 comments

Comments

@anthonyserious
Copy link

anthonyserious commented Oct 25, 2017

PR available at #523.

Querying ~40k points goes from CPU-pegged 71s to ~2s.

Before:

        1    0.039    0.039   71.545   71.545 <string>:1(<module>)
        1    0.008    0.008   68.592   68.592 client.py:306(_read_chunked_response)
        1    0.001    0.001   71.506   71.506 client.py:318(query)
       10   18.416    1.842   67.482    6.748 models.py:779(iter_lines)
    49125   48.934    0.001   48.934    0.001 {method 'splitlines' of 'str' objects}

After:

        1    0.034    0.034    6.527    6.527 <string>:1(<module>)
        1    0.002    0.002    3.912    3.912 client.py:310(_read_chunked_response)
        1    0.000    0.000    6.493    6.493 client.py:328(query)
      166    0.286    0.002    2.831    0.017 models.py:779(iter_lines)
    49299    2.487    0.000    2.487    0.000 {method 'splitlines' of 'str' objects}
@anthonyserious
Copy link
Author

Forgot to mention, above is with passing chunk_size=10000 vs. the default 512 in requests.

@sebito91
Copy link
Contributor

Closed by #753 and #538

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants