TST: mask DeprecationWarning in xfailed test #15610

mattip · 2020-02-19T21:01:51Z

xfailed tests are still run, they are expect to raise. This one was also emitting a DeprecationWarning, which is confusing

mattip · 2020-02-19T23:31:06Z

@r-devulap this is failing with reciprocal. After a deep dive, I found that the problem is that for certain values of the jj stride, especially -1, the IS_OUTPUT_BLOCKABLE_UNARY macro fails, we revert to glibc's implementation, and sometimes (especially for reciprocal) it is an ULP or two off. Can you mock the IS_OUTPUT_BLOCKABLE_UNARY macro and skip the assert_equal if it will fail?

seberg · 2020-02-19T23:54:10Z

@mattip what deprecation warning is this giving? I do not quite understand. Clip used to include a deprecation warning, which should never be hit. But that is now even moved the dtype creation phase, so what is going on?

r-devulap · 2020-02-20T01:27:03Z

hmm this is little bizarre. I can't reproduce this error locally on the manylinux2010 docker despite running 10,000 times and it doesn't seem like it fails on the CI consistently too. Reciprocal function is just a divide and should produce exactly the same result no matter what SIMD variant it uses (including the scalar code). There should be no ULP error.

mattip · 2020-02-20T04:37:58Z

@r-devulap the process to run the test builds a wheel with a manylinux2010 docker and tests with the matthewbrett/trusty:32 docker. If you run with a debugger (which I couldn't successfully do) or add a print when the IS_OUTPUT_BLOCKABLE_UNARY macro fails (which is what I ended up doing) you will see the macro does not succeed for all strides. Any failure of that macro means that part of the test is not checking what you want it to check, and so should be considered a failure.

If you want to reproduce the exact process the build follows, you can use bash buildscript.sh where buildscript.sh is below, but I think your time would be better spent thinking when the macro would fail, and avoid those cases.

``` # Once off, modify the numpy-wheel repo # Add the following to `config.sh` to use the current state of # the numpy submodule, otherwise it uses the git commit in # `build_commit` function clean_code { echo skipping git update } # Make the following changes to multibuild diff --git a/docker_test_wrap.sh b/docker_test_wrap.sh index a37b7e0..7305252 100755 --- a/docker_test_wrap.sh +++ b/docker_test_wrap.sh @@ -1,6 +1,7 @@ #!/bin/bash # Install and test steps on Linux -set -e +set -ex +export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'

Get needed utilities

MULTIBUILD_DIR=$(dirname "${BASH_SOURCE[0]}")
diff --git a/travis_linux_steps.sh b/travis_linux_steps.sh
index 0fc6a98..863e23a 100644
--- a/travis_linux_steps.sh
+++ b/travis_linux_steps.sh
@@ -11,7 +11,6 @@

install_run

set -e

-MANYLINUX_URL=${MANYLINUX_URL:-https://5cf40426d9f06eb7461d-6fe47d9331aba7cd62fc36c7196769e4.ssl.cf2.rackcdn.com}

Default Manylinux version

Warning: ignored if DOCKER_IMAGE variable is set.

@@ -19,7 +18,7 @@ MANYLINUX_URL=${MANYLINUX_URL:-https://5cf40426d9f06eb7461d-6fe47d9331aba7cd62fc
MB_ML_VER=${MB_ML_VER:-1}

Get our own location on this filesystem

-MULTIBUILD_DIR=$(dirname "${BASH_SOURCE[0]}")
+MULTIBUILD_DIR=multibuild

Allow travis Python version as proxy for multibuild Python version

MB_PYTHON_VERSION=${MB_PYTHON_VERSION:-$TRAVIS_PYTHON_VERSION}


This is the content of `buildscript.sh`

cd numpy-wheel
#remove previous run's wheel
sudo rm -rf wheelhouse/*
export repo_dir=numpy
export build_commit=e94cec800304a6a467cf90ce4e7d3e207770b4b4
export build_depends=cython
export test_depends="pytest hypothesis"
export plat=x86_64
export unicode_width=32
export wheelhouse_uploader_username=travis-worker
export daily_commit=master
export extra_argv="'--disable-pytest-warnings'"
export mb_python_version=3.7
export mb_ml_ver=2010
export plat=i686
export env_vars_path=env_vars_32.sh

set -x
source multibuild/common_utils.sh
source multibuild/travis_steps.sh

this uses a manylinux2010 docker to build the wheel

build_wheel $repo_dir $plat
sudo rm -rf tmp_for_test

This uses the other docker to install and test the wheel

install_run $plat

mattip · 2020-02-20T04:43:17Z

@seberg I convinced myself by adding a xxxx = xxxx failure line (or import pdb; pdb.set_trace()) in the code where the python code emits the warning and run this test only python runtest -- -k test_clip_scalar_nan_propagation. For me it hit hit the warning.

r-devulap · 2020-02-20T05:20:02Z

@mattip I am not sure what is causing this failure, but the ULP error seems very small (< 1 ULP, max relative error of 1.1176388e-16). May be using assert_array_max_ulp instead of assert_equal might be more suitable? Submitted a PR with this change #15615. Do you think that might suffice?

seberg · 2020-02-20T05:52:50Z

Sorry matti, nvm. I was confused, because I checked a non-development version and thought there was no datetime/timedelta loop for np.isnan. But we have one now, so this makes perfect sense.

mattip · 2020-02-20T06:17:20Z

Please wait until gh-15615 is settled before merging this, the CI failure here is interesting and difficult to reproduce, and I think merging it will make the log dissapear.

charris · 2020-02-20T18:48:15Z

numpy/core/tests/test_numeric.py

@@ -2018,7 +2018,8 @@ def test_NaT_propagation(self, arr, amin, amax):
        # NOTE: the expected function spec doesn't
        # propagate NaT, but clip() now does
        expected = np.minimum(np.maximum(arr, amin), amax)
-        actual = np.clip(arr, amin, amax)
+        with assert_warns(DeprecationWarning):
+            actual = np.clip(arr, amin, amax)


How about @pytest.mark.filterwarnings("ignore::DeprecationWarning")? Using assert_warns seems out of place for something that will be changed in the future, tests for warnings belong in test_deprecations.

Done. Another option is to delete this test: I think it started out as an edge case reported by hypothesis, and has now both an xfail and a warning filter.

charris · 2020-02-23T19:03:52Z

Thanks Matti.

mattip added the 05 - Testing label Feb 19, 2020

mattip mentioned this pull request Feb 19, 2020

ENH: expose bit_generator and random C-API to cython #15463

Merged

r-devulap mentioned this pull request Feb 20, 2020

BUG: Remove check requiring natural alignment of float/double to run AVX code #15615

Merged

charris reviewed Feb 20, 2020

View reviewed changes

mattip mentioned this pull request Feb 21, 2020

update submodules to latest versions MacPython/numpy-wheels#73

Merged

mattip force-pushed the xfail-warning branch from 08b51ba to 5b4dabe Compare February 23, 2020 08:52

TST: mask DeprecationWarning in xfailed tests

10bba3c

mattip force-pushed the xfail-warning branch from 5b4dabe to 10bba3c Compare February 23, 2020 18:09

charris merged commit 124dc4e into numpy:master Feb 23, 2020

mattip deleted the xfail-warning branch November 2, 2020 08:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST: mask DeprecationWarning in xfailed test #15610

TST: mask DeprecationWarning in xfailed test #15610

mattip commented Feb 19, 2020

mattip commented Feb 19, 2020 •

edited

Loading

seberg commented Feb 19, 2020

r-devulap commented Feb 20, 2020 •

edited

Loading

mattip commented Feb 20, 2020

Get needed utilities

install_run

Default Manylinux version

Warning: ignored if DOCKER_IMAGE variable is set.

Get our own location on this filesystem

Allow travis Python version as proxy for multibuild Python version

this uses a manylinux2010 docker to build the wheel

This uses the other docker to install and test the wheel

mattip commented Feb 20, 2020

r-devulap commented Feb 20, 2020

seberg commented Feb 20, 2020

mattip commented Feb 20, 2020

charris Feb 20, 2020

mattip Feb 23, 2020

charris commented Feb 23, 2020

TST: mask DeprecationWarning in xfailed test #15610

TST: mask DeprecationWarning in xfailed test #15610

Conversation

mattip commented Feb 19, 2020

mattip commented Feb 19, 2020 • edited Loading

seberg commented Feb 19, 2020

r-devulap commented Feb 20, 2020 • edited Loading

mattip commented Feb 20, 2020

Get needed utilities

install_run

Default Manylinux version

Warning: ignored if DOCKER_IMAGE variable is set.

Get our own location on this filesystem

Allow travis Python version as proxy for multibuild Python version

this uses a manylinux2010 docker to build the wheel

This uses the other docker to install and test the wheel

mattip commented Feb 20, 2020

r-devulap commented Feb 20, 2020

seberg commented Feb 20, 2020

mattip commented Feb 20, 2020

charris Feb 20, 2020

Choose a reason for hiding this comment

mattip Feb 23, 2020

Choose a reason for hiding this comment

charris commented Feb 23, 2020

mattip commented Feb 19, 2020 •

edited

Loading

r-devulap commented Feb 20, 2020 •

edited

Loading