Skip to content

enable cloud fetch by default #287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 2, 2025
Merged

Conversation

shivam2680
Copy link
Contributor

@shivam2680 shivam2680 commented Apr 1, 2025

Description

Enable cloud fetch (EXTERNAL_LINKS) by default.

Testing

Benchmark tests

Benchmarking results

test_case cloud fetch OFF cloud fetch ON
large_results_0500mb 203107 185524
large_results_0250mb 95478.29 88751.57
large_results_0100mb 36448.71 34643.57
large_results_0050mb 18533.29 18314.71
large_results_0025mb 10274.71 10258.00
large_results_0010mb 3646.57 3691.00
large_results_0005mb 1974.71 2167.43

We get ~9% deduction in large query execution time and this gap is bound to increase with increase in result size, thus, justifying enabling cloud fetch by default.

@@ -17,6 +17,7 @@ export type ExecuteStatementOptions = {
runAsync?: boolean;
maxRows?: number | bigint | Int64 | null;
useCloudFetch?: boolean;
useLZ4Compression?: boolean;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain why did you introduce this useLZ4Compression option in this PR? it does not seem relevant to cloudFetch enable by default?

Copy link
Contributor Author

@shivam2680 shivam2680 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can add this in a separate PR also. Just wanted to introduce compression, as this is configurable in other drivers.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is better to do this in a separate PR

@shivam2680 shivam2680 changed the title enable cloud fetch by default and make compression configurable enable cloud fetch by default Apr 1, 2025
@shivam2680 shivam2680 merged commit f357caa into main Apr 2, 2025
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants