Skip to content

[datalabeling] manage_dataset_test.test_list_dataset failed with DeadlineExceeded #3703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tmatsuo opened this issue May 7, 2020 · 2 comments
Closed

[datalabeling] manage_dataset_test.test_list_dataset failed with DeadlineExceeded #3703

tmatsuo opened this issue May 7, 2020 · 2 comments
Assignees
@tmatsuo
Labels
api: datalabeling Issues related to the AI Platform Data Labeling Service API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@tmatsuo
Copy link
Contributor

tmatsuo commented May 7, 2020

Build

From reading the invocation log, there seems to be many datasets. My suspicion is that now the project has many "datasets" which makes list_datasets to be more flaky. I think the function is fetching pages and pages under the cover.

I think manually deleting those datasets will greatly mitigate

Log:

Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1588849942.771695832","description":"Error received from peer ipv4:108.177.98.81:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/datalabeling/manage_dataset_test.py", line 72, in test_list_dataset
    run_sample()
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/datalabeling/manage_dataset_test.py", line 70, in run_sample
    manage_dataset.list_datasets(PROJECT_ID)
  File "/tmpfs/src/github/python-docs-samples/datalabeling/manage_dataset.py", line 73, in list_datasets
    for element in response:
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 212, in _items_iter
    for page in self._page_iter(increment=False):
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 249, in _page_iter
    page = self._next_page()
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 534, in _next_page
    response = self._method(self._request)
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 206, in retry_target
    last_exc,
  File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 600.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x7f44b77f1320>, parent: "projects/python-docs-samples-tests"
page_token: "2020-02-26T00:23:46.026853412-08:00"
, metadata=[('x-goog-request-params', 'parent=projects/python-docs-samples-tests'), ('x-goog-api-client', 'gl-python/3.7.5 grpc/1.28.1 gax/1.17.0 gapic/0.4.0')]), last exception: 504 Deadline Exceeded
@tmatsuo tmatsuo added priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: datalabeling Issues related to the AI Platform Data Labeling Service API. labels May 7, 2020
@tmatsuo tmatsuo self-assigned this May 7, 2020
@tmatsuo
Copy link
Contributor Author

tmatsuo commented May 7, 2020

For the record, there was 1943 datasets in the test project. Now I'm deleting them. I'll also add a PR to automatically delete stale datasets before the test run.

@tmatsuo tmatsuo mentioned this issue May 7, 2020
Merged
@tmatsuo
Copy link
Contributor Author

tmatsuo commented May 8, 2020

Now it's tracked by buildcop

@tmatsuo tmatsuo closed this as completed May 8, 2020
tmatsuo pushed a commit to tmatsuo/python-docs-samples that referenced this issue May 14, 2020
[datalabeling] fix: clean up old datasets before the test
d218629 
fixes GoogleCloudPlatform#3703
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tmatsuo tmatsuo

Labels
api: datalabeling Issues related to the AI Platform Data Labeling Service API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tmatsuo