Skip to content

fix: Use except distinct and intersect distinct #1094

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 18, 2024

Conversation

aholyoke
Copy link
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

According to BQ docs, the syntax for except and intersect is EXCEPT DISTINCT and INTERSECT DISTINCT respectively.

For example,

import sqlalchemy as sa

engine = sa.create_engine("bigquery://")
conn = engine.connect()

meta = sa.MetaData()
table = sa.Table(
    "set_example",
    meta,
    sa.Column("id", sa.Integer()),
    sa.Column("value", sa.Integer()),
    schema=DATASET_NAME,
)
table.create(bind=conn)

conn.execute(table.insert().values([
    {"id": 1, "value": 5},
    {"id": 2, "value": 2},
    {"id": 3, "value": 4},
    {"id": 4, "value": 9},
    {"id": 5, "value": 7},
    {"id": 6, "value": 1},
]))

expr = sa.select(table.c.id).except_(
    sa.select(table.c.id).where(table.c.value >= 5),
)
conn.execute(expr).fetchall()

Before this change we get:

DatabaseError: (google.cloud.bigquery.dbapi.exceptions.DatabaseError) 400 EXCEPT must be followed by ALL, DISTINCT, or "(" at [2:35]

[SQL: SELECT `set_example`.`id`
FROM `holyoke_test`.`set_example` EXCEPT SELECT `set_example`.`id`
FROM `holyoke_test`.`set_example`
WHERE `set_example`.`value` >= %(value_1:INT64)s]
[parameters: {'value_1': 5}]
(Background on this error at: https://sqlalche.me/e/14/4xp6)

After this change we get:
[(2,), (3,), (6,)]

Similarly, with

expr = sa.select(table.c.id).intersect(
    sa.select(table.c.id).where(table.c.value >= 5),
)
conn.execute(expr).fetchall()

Before this change we get:

DatabaseError: (google.cloud.bigquery.dbapi.exceptions.DatabaseError) 400 Syntax error: Expected keyword ALL or keyword DISTINCT but got keyword SELECT at [2:45]

[SQL: SELECT `set_example`.`id`
FROM `holyoke_test`.`set_example` INTERSECT SELECT `set_example`.`id`
FROM `holyoke_test`.`set_example`
WHERE `set_example`.`value` >= %(value_1:INT64)s]
[parameters: {'value_1': 5}]
(Background on this error at: https://sqlalche.me/e/14/4xp6)

After this change we get:
[(5,), (1,), (4,)]

It seems that this was discussed here with the maintainers of SA: sqlalchemy/sqlalchemy#11545, and the recommendation is for the dialect to implement the correct compound keywords for except_ and intersect

@aholyoke aholyoke requested review from a team as code owners July 13, 2024 21:01
@aholyoke aholyoke requested a review from hongalex July 13, 2024 21:01
@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. labels Jul 13, 2024
@Linchin Linchin assigned Linchin and unassigned farhan0102 Jul 18, 2024
@Linchin Linchin added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 18, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 18, 2024
@Linchin Linchin added the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 18, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 18, 2024
Copy link
Contributor

@Linchin Linchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for fixing this error! LGTM.

@Linchin Linchin merged commit 80781ef into googleapis:main Jul 18, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. size: m Pull request size is medium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants