Skip to content

BUG: Test failure on ppc64le #16955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
h-vetinari opened this issue Apr 18, 2020 · 1 comment
Open

BUG: Test failure on ppc64le #16955

h-vetinari opened this issue Apr 18, 2020 · 1 comment
Labels
Bug help wanted module:test-suite everything related to our tests

Comments

@h-vetinari
Copy link

h-vetinari commented Apr 18, 2020

Describe the bug

While working on conda-forge/scikit-learn-feedstock#123, I found that manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] fails on ppc64le; @rth asked me to open a bug here.

=========================== short test summary info ============================
FAILED manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] - Assertio...
= 1 failed, 13718 passed, 398 skipped, 6 xfailed, 1674 warnings in 382.15s (0:06:22) =

Here are the logs, the failure is the same across cpython 3.6-3.8:

Interestingly, PyPy doesn't have that failure (but has some segfaults instead..., see #16956).

A more detailed trace is below the fold. From what I can see from the log, the failure seems to be a question of "only" the quality of an embedding, and not a hard failure per se.

=================================== FAILURES ===================================
________________________ test_uniform_grid[barnes_hut] _________________________
[gw2] linux -- Python 3.6.10 $PREFIX/bin/python

method = 'barnes_hut'

    @pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
    def test_uniform_grid(method):
        """Make sure that TSNE can approximately recover a uniform 2D grid
    
        Due to ties in distances between point in X_2d_grid, this test is platform
        dependent for ``method='barnes_hut'`` due to numerical imprecision.
    
        Also, t-SNE is not assured to converge to the right solution because bad
        initialization can lead to convergence to bad local minimum (the
        optimization problem is non-convex). To avoid breaking the test too often,
        we re-run t-SNE from the final point when the convergence is not good
        enough.
        """
        seeds = [0, 1, 2]
        n_iter = 500
        for seed in seeds:
            tsne = TSNE(n_components=2, init='random', random_state=seed,
                        perplexity=20, n_iter=n_iter, method=method)
            Y = tsne.fit_transform(X_2d_grid)
    
            try_name = "{}_{}".format(method, seed)
            try:
>               assert_uniform_grid(Y, try_name)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:784: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Y = array([[ 52.326397  , -15.92225   ],
       [ 46.679527  , -20.175953  ],
       [ 40.870537  , -24.181147  ],
       ...[-35.291374  ,  22.122814  ],
       [-42.2738    ,  18.793724  ],
       [-48.922283  ,  15.606232  ]], dtype=float32)
try_name = 'barnes_hut_1'

    def assert_uniform_grid(Y, try_name=None):
        # Ensure that the resulting embedding leads to approximately
        # uniformly spaced points: the distance to the closest neighbors
        # should be non-zero and approximately constant.
        nn = NearestNeighbors(n_neighbors=1).fit(Y)
        dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
        assert dist_to_nn.min() > 0.1
    
        smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
        largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
    
        assert smallest_to_mean > .5, try_name
>       assert largest_to_mean < 2, try_name
E       AssertionError: barnes_hut_1
E       assert 6.67359409617653 < 2

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError

During handling of the above exception, another exception occurred:

method = 'barnes_hut'

    @pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
    def test_uniform_grid(method):
        """Make sure that TSNE can approximately recover a uniform 2D grid
    
        Due to ties in distances between point in X_2d_grid, this test is platform
        dependent for ``method='barnes_hut'`` due to numerical imprecision.
    
        Also, t-SNE is not assured to converge to the right solution because bad
        initialization can lead to convergence to bad local minimum (the
        optimization problem is non-convex). To avoid breaking the test too often,
        we re-run t-SNE from the final point when the convergence is not good
        enough.
        """
        seeds = [0, 1, 2]
        n_iter = 500
        for seed in seeds:
            tsne = TSNE(n_components=2, init='random', random_state=seed,
                        perplexity=20, n_iter=n_iter, method=method)
            Y = tsne.fit_transform(X_2d_grid)
    
            try_name = "{}_{}".format(method, seed)
            try:
                assert_uniform_grid(Y, try_name)
            except AssertionError:
                # If the test fails a first time, re-run with init=Y to see if
                # this was caused by a bad initialization. Note that this will
                # also run an early_exaggeration step.
                try_name += ":rerun"
                tsne.init = Y
                Y = tsne.fit_transform(X_2d_grid)
>               assert_uniform_grid(Y, try_name)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:792: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Y = array([[-18.169476  ,   6.0802336 ],
       [-18.278513  ,   2.8822129 ],
       [-18.671782  ,  -0.4646889 ],
       ...[ 22.550077  ,  19.698557  ],
       [ 21.399723  ,  22.933178  ],
       [ 16.22136   ,  28.22955   ]], dtype=float32)
try_name = 'barnes_hut_1:rerun'

    def assert_uniform_grid(Y, try_name=None):
        # Ensure that the resulting embedding leads to approximately
        # uniformly spaced points: the distance to the closest neighbors
        # should be non-zero and approximately constant.
        nn = NearestNeighbors(n_neighbors=1).fit(Y)
        dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
        assert dist_to_nn.min() > 0.1
    
        smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
        largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
    
        assert smallest_to_mean > .5, try_name
>       assert largest_to_mean < 2, try_name
E       AssertionError: barnes_hut_1:rerun
E       assert 2.145051767903112 < 2

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError
=============================== warnings summary ===============================
[...]
@isuruf
Copy link
Contributor

isuruf commented Apr 27, 2020

Duplicate of #15821

@cmarmo cmarmo added Bug module:test-suite everything related to our tests help wanted and removed Bug: triage Build / CI labels Sep 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug help wanted module:test-suite everything related to our tests
Projects
None yet
Development

No branches or pull requests

3 participants