-
-
Notifications
You must be signed in to change notification settings - Fork 25.8k
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil ⚠️ #24221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I can not reproduce this one locally, let's see if it happens consistently in the CI ... |
Note that this test also has For the record, here it fails only with:
|
I will manually retrigger this CI build in Azure to see if it's deterministic or not. Here is the traceback of the original failure: ________________________ test_kernel_gradient[kernel11] ________________________
[gw1] linux -- Python 3.9.10 /home/vsts/work/1/s/testvenv/bin/python
kernel = 1.41**2 * Matern(length_scale=[0.5, 2], nu=0.5)
@fails_if_pypy
@pytest.mark.parametrize("kernel", kernels)
def test_kernel_gradient(kernel):
# Compare analytic and numeric gradient of kernels.
K, K_gradient = kernel(X, eval_gradient=True)
assert K_gradient.shape[0] == X.shape[0]
assert K_gradient.shape[1] == X.shape[0]
assert K_gradient.shape[2] == kernel.theta.shape[0]
def eval_kernel_for_theta(theta):
kernel_clone = kernel.clone_with_theta(theta)
K = kernel_clone(X, eval_gradient=False)
return K
K_gradient_approx = _approx_fprime(kernel.theta, eval_kernel_for_theta, 1e-10)
> assert_almost_equal(K_gradient, K_gradient_approx, 4)
E AssertionError:
E Arrays are not almost equal to 4 decimals
E
E Mismatched elements: 8 / 75 (10.7%)
E Max absolute difference: 71.55003086
E Max relative difference: 0.67469822
E x: array([[[2.0000e+000, 1.3853e-309, 9.3709e-310],
[2.38801479e-01, 4.98647235e-01, 8.87095952e-03],
[2.00000017e+00, 0.00000000e+00, 0.00000000e+00]]])
eval_kernel_for_theta = <function test_kernel_gradient.<locals>.eval_kernel_for_theta at 0x4c915408d50>
kernel = 1.41**2 * Matern(length_scale=[0.5, 2], nu=0.5)
|
CI is no longer failing! ✅Successful run on Aug 22, 2022 |
Ok, so apparently this not deterministic. It could be a race condition of some sort though but given that it's not easy to reproduce, I think we can close for now. If it shows up again we will give it a higher priority to try to investigate the root cause. |
I looked a bit more into this one, I can reproduce the failure with pypy. We are using code like: result = np.divide(numerator, denominator, where=denominator != 0)
A snippet to reproduce a similar behaviour: import numpy as np
numerator = np.array([1., 1., 1.])
denominator = np.array([0., 1., 1.])
where = denominator != 0
for i in range(5):
# this temporary variable impacts the values for denominator == 0 ...
# probably by reusing the memory :-(
tmp = numerator * (i + denominator)
print(f"tmp address: {tmp.__array_interface__['data'][0]}")
del tmp
divide_result = np.divide(numerator, denominator, where=where)
print(f"div address: {divide_result.__array_interface__['data'][0]}")
print(f"{divide_result}") Output with CPython:
Output with pypy:
|
Opened #24245 to fix the problem for |
CI failed on Linux_nogil.pylatest_pip_nogil (Aug 22, 2022)
The text was updated successfully, but these errors were encountered: