Skip to content

IN filter causes Database syntax error if printed before execution #357

Closed
@LocalFolioProduct

Description

@LocalFolioProduct

Environment details

  • OS type and version: macOS Big Sur 11.6
  • Python version: Python 3.9.5
  • pip version: 21.2.4
  • sqlalchemy-bigquery version: 1.2.0

Steps to reproduce

  1. Create Query containing IN filter
  2. Call Query.__str__() (e.g. print(Query), log(Query))
  3. Execute query
  4. DatabaseError 400 Syntax error raised

Code example

from sqlalchemy import String, Float, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy.schema import Column

from bq_reporting import settings

Base = declarative_base()
BIG_QUERY_PROJECT = 'BIG_QUERY_PROJECT'

class BlockGroup(Base):
    __tablename__ = f'bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr'

    geo_id = Column(String, primary_key=True)
    nonfamily_households = Column(Float)


engine = create_engine(f'bigquery://{BIG_QUERY_PROJECT}')
session = sessionmaker(bind=engine)()

results = session.query(
    BlockGroup.geo_id
).filter(BlockGroup.geo_id.in_(['213213']))
print(results)
print(results.count())

Stack trace

SELECT `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` AS `bigquery_public_data_census_bureau_acs_blockgroup_2010_5yr_geo_id` 
FROM `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr` 
WHERE `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` IN UNNEST(%(geo_id_1:STRING)s)
Traceback (most recent call last):
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 203, in _execute
    self._query_job.result()
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1372, in result
    do_get_result()
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1362, in do_get_result
    super(QueryJob, self).result(retry=retry, timeout=timeout)
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 708, in result
    return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/api_core/future/polling.py", line 137, in result
    raise self._exception
google.api_core.exceptions.BadRequest: 400 Syntax error: Expected "(" or keyword UNNEST but got "@" at [4:80]

(job ID: 60c732c3-4b4e-4da6-9988-18ba20d389ee)

                                                                -----Query Job SQL Follows-----                                                                

    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |
   1:SELECT count(*) AS `count_1` 
   2:FROM (SELECT `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` AS `bigquery_public_data_census_bureau_acs_blockgroup_2010_5yr_geo_id` 
   3:FROM `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr` 
   4:WHERE `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` IN @`geo_id_1`) AS `anon_1`
    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
    self.dialect.do_execute(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 489, in with_closed_check
    return method(self, *args, **kwargs)
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 166, in execute
    self._execute(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 205, in _execute
    raise exceptions.DatabaseError(exc)
google.cloud.bigquery.dbapi.exceptions.DatabaseError: 400 Syntax error: Expected "(" or keyword UNNEST but got "@" at [4:80]

(job ID: 60c732c3-4b4e-4da6-9988-18ba20d389ee)

                                                                -----Query Job SQL Follows-----                                                                

    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |
   1:SELECT count(*) AS `count_1` 
   2:FROM (SELECT `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` AS `bigquery_public_data_census_bureau_acs_blockgroup_2010_5yr_geo_id` 
   3:FROM `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr` 
   4:WHERE `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` IN @`geo_id_1`) AS `anon_1`
    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/localuser/Library/Application Support/JetBrains/PyCharm2021.1/scratches/scratch_9.py", line 26, in <module>
    print(results.count())
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3090, in count
    return self._from_self(col).enable_eagerloads(False).scalar()
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2831, in scalar
    ret = self.one()
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2808, in one
    return self._iter().one()
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2846, in _iter
    result = self.session.execute(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1611, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1478, in _execute_clauseelement
    ret = self._execute_context(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
    self._handle_dbapi_exception(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2023, in _handle_dbapi_exception
    util.raise_(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
    self.dialect.do_execute(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 489, in with_closed_check
    return method(self, *args, **kwargs)
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 166, in execute
    self._execute(
  File "/Users/localuser/dev/venv/bq_reporting/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 205, in _execute
    raise exceptions.DatabaseError(exc)
sqlalchemy.exc.DatabaseError: (google.cloud.bigquery.dbapi.exceptions.DatabaseError) 400 Syntax error: Expected "(" or keyword UNNEST but got "@" at [4:80]

(job ID: 60c732c3-4b4e-4da6-9988-18ba20d389ee)

                                                                -----Query Job SQL Follows-----                                                                

    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |
   1:SELECT count(*) AS `count_1` 
   2:FROM (SELECT `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` AS `bigquery_public_data_census_bureau_acs_blockgroup_2010_5yr_geo_id` 
   3:FROM `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr` 
   4:WHERE `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` IN @`geo_id_1`) AS `anon_1`
    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |
[SQL: SELECT count(*) AS `count_1` 
FROM (SELECT `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` AS `bigquery_public_data_census_bureau_acs_blockgroup_2010_5yr_geo_id` 
FROM `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr` 
WHERE `bigquery-public-data.census_bureau_acs.blockgroup_2010_5yr`.`geo_id` IN %(geo_id_1:STRING)s) AS `anon_1`]
[parameters: {'geo_id_1': ['213213']}]
(Background on this error at: https://sqlalche.me/e/14/4xp6)

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the googleapis/python-bigquery-sqlalchemy API.priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions