Skip to content

KMeans()-check_transformer_data_not_an_array test failing locally from time to time in main #20199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
lesteve opened this issue Jun 3, 2021 · 2 comments · Fixed by #20200
Closed

KMeans()-check_transformer_data_not_an_array test failing locally from time to time in main #20199

lesteve opened this issue Jun 3, 2021 · 2 comments · Fixed by #20200

Comments

@lesteve
Copy link
Member

lesteve commented Jun 3, 2021 •
edited
Loading

Describe the bug

Locally I see intermittent failures of the KMeans()-check_transformer_data_not_an_array test. I don't see this failures on 0.24.2.

One additional weird thing is that this is not happening in the CI and I seem to be the first to complain about it (at least I could not find it in the issues).

❯ pytest sklearn/tests/test_common.py  -k 'KMeans and data_not_an_array'
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.7.7, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/lesteve/dev/scikit-learn/.hypothesis/examples')
rootdir: /home/lesteve/dev/scikit-learn, configfile: setup.cfg
plugins: hypothesis-4.36.2, asyncio-0.10.0, cov-2.7.1
collected 7785 items / 7783 deselected / 2 selected                                                                                                                                                                                                                                      

sklearn/tests/test_common.py F.                                                                                                                                                                                                                                                    [100%]

======================================================================================================================================== FAILURES ========================================================================================================================================
_____________________________________________________________________________________________________________ test_estimators[KMeans()-check_transformer_data_not_an_array] ______________________________________________________________________________________________________________

estimator = KMeans(max_iter=5, n_clusters=2, n_init=2), check = functools.partial(<function check_transformer_data_not_an_array at 0x7fec5ceb7050>, 'KMeans'), request = <FixtureRequest for <Function test_estimators[KMeans()-check_transformer_data_not_an_array]>>

    @parametrize_with_checks(list(_tested_estimators()))
    def test_estimators(estimator, check, request):
        # Common tests for estimator instances
        with ignore_warnings(category=(FutureWarning,
                                       ConvergenceWarning,
                                       UserWarning, FutureWarning)):
            _set_checking_parameters(estimator)
>           check(estimator)

sklearn/tests/test_common.py:90: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/utils/_testing.py:308: in wrapper
    return fn(*args, **kwargs)
sklearn/utils/estimator_checks.py:1289: in check_transformer_data_not_an_array
    _check_transformer(name, transformer, X, y)
sklearn/utils/estimator_checks.py:1366: in _check_transformer
    % transformer, atol=1e-2)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = array([[0.20255037, 3.57452256],
       [3.22900935, 0.23278803],
       [3.2526988 , 0.33507256],
       [0.34546063,...79101, 0.53812952],
       [0.32877832, 3.45131422],
       [0.19314137, 3.21278957],
       [3.79826855, 0.41606372]])
y = array([[3.57452256, 0.20255037],
       [0.23278803, 3.22900935],
       [0.33507256, 3.2526988 ],
       [3.23476868,...12952, 3.75679101],
       [3.45131422, 0.32877832],
       [3.21278957, 0.19314137],
       [0.41606372, 3.79826855]]), rtol = 1e-07, atol = 0.01
err_msg = 'fit_transform and transform outcomes not consistent in KMeans(max_iter=5, n_clusters=2, n_init=2, random_state=0)'

    def assert_allclose_dense_sparse(x, y, rtol=1e-07, atol=1e-9, err_msg=''):
        """Assert allclose for sparse and dense data.
    
        Both x and y need to be either sparse or dense, they
        can't be mixed.
    
        Parameters
        ----------
        x : {array-like, sparse matrix}
            First array to compare.
    
        y : {array-like, sparse matrix}
            Second array to compare.
    
        rtol : float, default=1e-07
            relative tolerance; see numpy.allclose.
    
        atol : float, default=1e-9
            absolute tolerance; see numpy.allclose. Note that the default here is
            more tolerant than the default for numpy.testing.assert_allclose, where
            atol=0.
    
        err_msg : str, default=''
            Error message to raise.
        """
        if sp.sparse.issparse(x) and sp.sparse.issparse(y):
            x = x.tocsr()
            y = y.tocsr()
            x.sum_duplicates()
            y.sum_duplicates()
            assert_array_equal(x.indices, y.indices, err_msg=err_msg)
            assert_array_equal(x.indptr, y.indptr, err_msg=err_msg)
            assert_allclose(x.data, y.data, rtol=rtol, atol=atol, err_msg=err_msg)
        elif not sp.sparse.issparse(x) and not sp.sparse.issparse(y):
            # both dense
>           assert_allclose(x, y, rtol=rtol, atol=atol, err_msg=err_msg)
E           AssertionError: 
E           Not equal to tolerance rtol=1e-07, atol=0.01
E           fit_transform and transform outcomes not consistent in KMeans(max_iter=5, n_clusters=2, n_init=2, random_state=0)
E           Mismatched elements: 60 / 60 (100%)
E           Max absolute difference: 3.38712923
E           Max relative difference: 24.28104678
E            x: array([[0.20255 , 3.574523],
E                  [3.229009, 0.232788],
E                  [3.252699, 0.335073],...
E            y: array([[3.574523, 0.20255 ],
E                  [0.232788, 3.229009],
E                  [0.335073, 3.252699],...

sklearn/utils/_testing.py:415: AssertionError
=============================================================================================================== 1 failed, 1 passed, 7783 deselected, 32 warnings in 2.69s ================================================================================================================

Steps/Code to Reproduce

I can reproduce this failure consistently with:

Create a test-kmeans.sh file:

#!/bin/bash
set -e

conda create -n test python scipy cython pytest joblib threadpoolctl -y
conda activate test
pip install --no-build-isolation --editable .

# make sure to run it a few times to trigger the test failure
for i in $(seq 1 50); do
    pytest sklearn/tests/test_common.py  -k 'KMeans and data_not_an_array'
done

Run test-kmeans.sh:

source test-kmeans.py

Expected Results

No test failure

Actual Results

Test failure

Other comments

Looking at bit at bit more, it seems that when calling KMeans.fit the cluster centers can be in a different order in main, whereas the order is consistent in 0.24.2. Wild-guess: maybe something due to some use of low-level parallelism in KMeans?

import numpy as np

from sklearn.cluster import KMeans

from sklearn.datasets import make_blobs

X, y = make_blobs(n_samples=30, centers=[[0, 0, 0], [1, 1, 1]],
                  random_state=0, n_features=2, cluster_std=0.1)

kmeans = KMeans(max_iter=5, n_clusters=2, n_init=2, random_state=0)

ref_cluster_centers = kmeans.fit(X, y).cluster_centers_
print('reference cluster centers:\n', ref_cluster_centers)
print('-'*80)

for i in range(100):
    cluster_centers = kmeans.fit(X, y).cluster_centers_
    if not np.allclose(ref_cluster_centers, cluster_centers):
        print('differing cluster centers:\n', cluster_centers)

Versions

System:
    python: 3.9.5 (default, May 18 2021, 19:34:48)  [GCC 7.3.0]
executable: /home/lesteve/miniconda3/envs/test/bin/python
   machine: Linux-5.4.0-73-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.1.1
   setuptools: 52.0.0.post20210125
      sklearn: 1.0.dev0
        numpy: 1.20.2
        scipy: 1.6.2
       Cython: 0.29.23
       pandas: None
   matplotlib: None
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
None
@lesteve
Copy link
Member Author

lesteve commented Jun 3, 2021 •
edited
Loading

I guess it would be nice if someone could reproduce first (maybe I have something special on my setup 🤔)

Otherwise git bisect points the finger at #17622. I am guessing @jeremiedbb may have some insights into this.

@jeremiedbb jeremiedbb mentioned this issue Jun 3, 2021
Merged
@jeremiedbb
Copy link
Member

I can reproduce locally as well. I opened #20200. It fixes it for me, let me know if it fixes it for you as well

@lesteve lesteve mentioned this issue Sep 27, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

More stable n_init runs for KMeans jeremiedbb/scikit-learn
2 participants
@lesteve @jeremiedbb