Skip to content

feat: Add "time_partitioning" and "clustering_fields" in table creation process #891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 13 commits into from
Closed

feat: Add "time_partitioning" and "clustering_fields" in table creation process #891

wants to merge 13 commits into from

Conversation

egormartiniuc
Copy link

@egormartiniuc egormartiniuc commented Jul 26, 2023 •
edited by chalmerlowe
Loading

Added "time_partitioning" and "clustering_fields" in table creation schema by keyword options "bigquery_time_partitioning" and "bigquery_clustering_fields".

creation sample:

sqlalchemy.Table(
        '{}.{}.{}'.format(project_id, dataset_id, table_id),
        sqlalchemy.MetaData(), 
       *columns,
        bigquery_time_partitioning=sqlalchemy_bigquery.TimePartitioning(
            sqlalchemy_bigquery.TimePartitioning.TimePartitioningType.DAY,
            "time", # time column name
            31536000000, # expiration in milliseconds default: 365 days 
            True # require_partition_filter default: True 
        ), 
        bigquery_clustering_fields=["A", "B"])

@egormartiniuc
Update base.py
f3ba1a7 
Added "time_partitioning" and "clustering_fields" in table creation schema by keyword options "bigquery_time_partitioning" and "bigquery_clustering_fields".
@egormartiniuc
Update __init__.py
2ccbdb0 
Added "TimePartitioning" class in __init__
@egormartiniuc egormartiniuc requested review from a team as code owners July 26, 2023 08:04
@egormartiniuc egormartiniuc requested a review from shollyman July 26, 2023 08:04
@google-cla
Copy link

google-cla bot commented Jul 26, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. labels Jul 26, 2023
@egormartiniuc
Update base.py
1e996db 
extended class "TimePartitioning" with conversion bigquery.TimePartitioning to sqlalchemy_bigquery.TimePartitioning by class function "frombigquery_time_partitioning"
@egormartiniuc
Update base.py
f7ce7d3 
Added basic check with rise of error if "Time & Cluster" columns not in the table.
@egormartiniuc
Update base.py
1f9f189 
Added export in the header

from sqlalchemy.exc import NoSuchColumnError
@parthea parthea added owlbot:run Add this label to trigger the Owlbot post processor. kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jul 26, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 26, 2023
@parthea parthea changed the title Added "time_partitioning" and "clustering_fields" in table creation process feat: Add "time_partitioning" and "clustering_fields" in table creation process Jul 26, 2023
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jul 26, 2023
@egormartiniuc
Update base.py
7f59036 
Removed "typing.Self" for compatibility with python < 3.11
@shollyman shollyman requested review from chalmerlowe and removed request for shollyman July 28, 2023 15:49
@leblancfg
Copy link

I've been keeping an eye on this PR ever since it's been put up. For most tables, not being able to specify partitioning and clustering is a non-starter for using this SQLAlchemy dialect.

Is there any way I might help to take it across the finish line? CC @chalmerlowe @tswast

@Linchin Linchin added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 6, 2023
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 7, 2023
@chalmerlowe
Copy link
Collaborator

@egormartiniuc
@leblancfg

There appear to be two PRs (this one and PR 928) that are aiming to accomplish similar outcomes. I am doing my best to examine both of them and see what makes the most sense in terms of next steps.

@chalmerlowe chalmerlowe self-assigned this Dec 19, 2023
@kiraksi kiraksi added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 29, 2024
@tswast
Copy link
Collaborator

tswast commented Jan 30, 2024

Closing this, as #928 has been merged.

@tswast tswast closed this Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chalmerlowe chalmerlowe Awaiting requested review from chalmerlowe

Assignees

@chalmerlowe chalmerlowe

Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. kokoro:force-run Add this label to force Kokoro to re-run the tests. size: m Pull request size is medium.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@egormartiniuc @leblancfg @chalmerlowe @tswast @parthea @Linchin @yoshi-kokoro @kiraksi