Skip to content

pyarrow dependency should be optional #743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
twm opened this issue Feb 2, 2023 · 5 comments · Fixed by #847
Closed

pyarrow dependency should be optional #743

twm opened this issue Feb 2, 2023 · 5 comments · Fixed by #847
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@twm
Copy link

twm commented Feb 2, 2023

pyarrow is an optional dependency of google-cloud-bigquery, but it's made mandatory by python-bigquery-sqlalchemy.

As pyarrow is quite large on disk — 100 MB on x86_64 Linux — I don't want to install it when it's unused in my application (AFAICT I don't need google-cloud-bigquery-storage either but that's not huge).

I suggest:

  1. Removing the direct dependencies on pyarrow and google-cloud-bigquery-storage
  2. Add a bqstorage extra that depends on google-cloud-bigquery[bqstorage]. That'll respect upstream's version bounds without introducing local bounds that could cause conflicts for users.
  3. Document that users wanting improved performance with large result sets should install bigquery-sqlalchemy[bqstorage].

There's an existing PR at #470 but it looks like it has stalled out, so I'm filing this issue to provide a blueprint for someone who's able to do this work.
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. label Feb 2, 2023
@shollyman shollyman added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Mar 3, 2023
@shollyman
Copy link

marking as a packaging-related issue for now, pending someone picking this up

@chalmerlowe chalmerlowe mentioned this issue Mar 17, 2023
Merged
@chalmerlowe
Copy link
Collaborator

@twm Thanks for creating this issue. Your suggestions were much appreciated and helped us move the ball down the field.
pyarrow and bqstorage should both be optional in the next release.

@twm
Copy link
Author

twm commented Mar 22, 2023

@chalmerlowe Awesome, glad to be of help! Thank you!

@jlynchMicron
Copy link

Hi All,

Now that bqstorage is an optional dependency, how does one specify using it (other than installing the dependencies)? If you install "sqlalchemy-bigquery[bqstorage]", will all queries automatically be ran with the bqstorage API? From browsing around the code, my guess is no and this issue (googleapis/python-bigquery#757) mentions it now possibly being a no-op.

At this point, is bqstorage support completely dropped from sqlalchemy-bigquery?

@parthea
Copy link
Contributor

parthea commented Dec 13, 2023

Hi @jlynchMicron,
Please could you open a new issue with your question?

cc @chalmerlowe

@jlynchMicron jlynchMicron mentioned this issue Dec 13, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Remove pyarrow and bqstorage as dependencies googleapis/python-bigquery-sqlalchemy
5 participants
@twm @shollyman @parthea @chalmerlowe @jlynchMicron