DISABLED test_fn_grad_grid_sampler_2d_cuda_float64 (__main__.TestBwdGradientsCUDA) #131079
Description
Platforms: inductor, rocm
This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.
Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.
Debugging instructions (after clicking on the recent samples link):
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
- Click on the workflow logs linked above
- Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
- Grep for
test_fn_grad_grid_sampler_2d_cuda_float64 - There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.
Sample error message
Traceback (most recent call last):
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 1127, in test_wrapper
return test(*args, **kwargs)
File "test_ops_gradients.py", line 37, in test_fn_grad
self._grad_test_helper(device, dtype, op, op.get_op())
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 5044, in _grad_test_helper
return self._check_helper(device, dtype, op, variant, 'gradcheck', check_forward_ad=check_forward_ad,
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 5013, in _check_helper
self.assertTrue(gradcheck(fn, gradcheck_args,
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 4583, in gradcheck
return torch.autograd.gradcheck(fn, inputs, **kwargs)
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 2053, in gradcheck
return _gradcheck_helper(**args)
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 2082, in _gradcheck_helper
_gradcheck_real_imag(
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 1492, in _gradcheck_real_imag
gradcheck_fn(
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 1922, in _fast_gradcheck
analytical_vJu = _get_analytical_vJu_backward_mode(
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 805, in _get_analytical_vJu_backward_mode
all_vJ = _check_analytical_jacobian_attributes(
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 791, in _check_analytical_jacobian_attributes
raise GradcheckError(
torch.autograd.gradcheck.GradcheckError: Backward is not reentrant, i.e., running backward with same input and grad_output multiple times gives different values, although analytical gradient matches numerical gradient.The tolerance for nondeterminism was 1e-15.
NOTE: If your op relies on non-deterministic operations i.e., it is listed here:
https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html
this failure might be expected.
If you are adding a new operator, please file an issue and then use one of the
workarounds. The workaround depends on how your test invokes gradcheck/gradgradcheck.
If the test
- manually invokes gradcheck/gradgradcheck, then call gradcheck/gradgradcheck
with `nondet_tol=<tol>` as a keyword argument.
- is OpInfo-based (e.g., in test_ops_gradients.py), then modify the OpInfo for the test
to have `gradcheck_nondet_tol=<tol>`.
- is a Module test (e.g., in common_nn.py), then modify the corresponding
module_test entry to have `gradcheck_nondet_tol=<tol>`
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 2829, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 2829, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 447, in instantiated_test
result = test(self, **param_kwargs)
File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 1139, in test_wrapper
raise e_tracked from e
Exception: Caused by sample input at index 14: SampleInput(input=Tensor[size=(2, 3, 5, 20), device="cuda:0", dtype=torch.float64], args=(Tensor[size=(2, 13, 10, 2), device="cuda:0", dtype=torch.float64],2,1,False), kwargs={}, broadcasts_input=False, name='')
To execute this test, run the following from the base repo dir:
PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=14 PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_WITH_INDUCTOR=1 python test/test_ops_gradients.py -k TestBwdGradientsCUDA.test_fn_grad_grid_sampler_2d_cuda_float64
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Test file path: test_ops_gradients.py