Open
Description
As stated in the issue title, if access token is expired / invalid, but refresh token is still valid, requests fail with google.api_core.exceptions.Unauthenticated: 401 Request had invalid authentication credentials. This could happen if the customer's clock is out-of-sync, for example.
This issue is in response to a customer issue (internal tracker 191460918).
Environment details
- OS: macOS
- Python version: Python 3.9.5
- pip version: pip 21.1.3
google-authversion:
$ conda list | grep google
google-api-core 1.30.0 pyhd8ed1ab_0 conda-forge
google-api-core-grpc 1.30.0 hd8ed1ab_0 conda-forge
google-auth 1.32.0 pyh6c4a22f_0 conda-forge
google-cloud-bigquery 2.20.0 pyhd3deb0d_0 conda-forge
google-cloud-bigquery-core 2.20.0 pyhd3deb0d_0 conda-forge
google-cloud-bigquery-storage-core 2.2.1 pyh44b312d_0 conda-forge
google-cloud-core 1.7.1 pyh6c4a22f_0 conda-forge
google-crc32c 1.1.2 py39he650545_0 conda-forge
google-resumable-media 1.3.1 pyh6c4a22f_0 conda-forge
googleapis-common-protos 1.53.0 py39h6e9494a_0 conda-forge
Steps to reproduce
Works fine:
from google.cloud import bigquery_storage
from google.cloud.bigquery_storage import types
import google.auth
import google.auth.transport.requests
creds, _ = google.auth.default()
creds.refresh(google.auth.transport.requests.Request())
creds.token = "DEFINITELY_EXPIRED"
creds.refresh(google.auth.transport.requests.Request())
session = google.auth.transport.requests.AuthorizedSession(creds)
bqstorageclient = bigquery_storage.BigQueryReadClient(credentials=creds)
project_id = "bigquery-public-data"
dataset_id = "new_york_trees"
table_id = "tree_species"
table = f"projects/{project_id}/datasets/{dataset_id}/tables/{table_id}"
parent = "projects/{}".format(PROJECT_ID)
requested_session = types.ReadSession(
table=table,
data_format=types.DataFormat.ARROW,
)
read_session = bqstorageclient.create_read_session(
parent=parent, read_session=requested_session, max_stream_count=1,
)
print(read_session.streams[0])Fails:
from google.cloud import bigquery_storage
from google.cloud.bigquery_storage import types
import google.auth
import google.auth.transport.requests
creds, _ = google.auth.default()
creds.refresh(google.auth.transport.requests.Request())
creds.token = "DEFINITELY_EXPIRED"
# creds.refresh(google.auth.transport.requests.Request()) <<< No forced refresh after token expiration.
session = google.auth.transport.requests.AuthorizedSession(creds)
bqstorageclient = bigquery_storage.BigQueryReadClient(credentials=creds)
project_id = "bigquery-public-data"
dataset_id = "new_york_trees"
table_id = "tree_species"
table = f"projects/{project_id}/datasets/{dataset_id}/tables/{table_id}"
parent = "projects/{}".format(PROJECT_ID)
requested_session = types.ReadSession(
table=table,
data_format=types.DataFormat.ARROW,
)
read_session = bqstorageclient.create_read_session(
parent=parent, read_session=requested_session, max_stream_count=1,
)
print(read_session.streams[0])Stack trace:
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/envs/dev-3.9/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/dev-3.9/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/Caskroom/miniconda/base/envs/dev-3.9/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."
debug_error_string = "{"created":"@1625004024.706701000","description":"Error received from peer ipv6:[2607:f8b0:4009:818::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.","grpc_status":16}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/swast/src/scratch/2021/06-b191460918-read-api/access_token_experiment.py", line 37, in <module>
read_session = bqstorageclient.create_read_session(
File "/usr/local/Caskroom/miniconda/base/envs/dev-3.9/lib/python3.9/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/client.py", line 508, in create_read_session
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/usr/local/Caskroom/miniconda/base/envs/dev-3.9/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/dev-3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 285, in retry_wrapped_func
return retry_target(
File "/usr/local/Caskroom/miniconda/base/envs/dev-3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 188, in retry_target
return target()
File "/usr/local/Caskroom/miniconda/base/envs/dev-3.9/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.Unauthenticated: 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
Desired fix
Refresh and retry after 401 errors. This is subtly different from adding Unauthenticated to the default list of retryable errors in that a refresh should be attempted first.
Perhaps this issue is best addressed in google-api-core?