[MRG] Make csgraph imports explicit #10145

lesteve · 2017-11-15T16:34:11Z

with scipy 1.0, pytest sklearn/semi_supervised give errors:

(test) ❯ pytest sklearn/semi_supervised
========================================== test session starts ===========================================
platform linux -- Python 3.6.3, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /home/lesteve/dev/scikit-learn, inifile: setup.cfg
collected 13 items                                                                                        

sklearn/semi_supervised/label_propagation.py ..F
sklearn/semi_supervised/tests/test_label_propagation.py FFFF.F.FFF

================================================ FAILURES ================================================
___________________ [doctest] sklearn.semi_supervised.label_propagation.LabelSpreading ___________________
474     --------
475     >>> from sklearn import datasets
476     >>> from sklearn.semi_supervised import LabelSpreading
477     >>> label_prop_model = LabelSpreading()
478     >>> iris = datasets.load_iris()
479     >>> rng = np.random.RandomState(42)
480     >>> random_unlabeled_points = rng.rand(len(iris.target)) < 0.3
481     >>> labels = np.copy(iris.target)
482     >>> labels[random_unlabeled_points] = -1
483     >>> label_prop_model.fit(iris.data, labels)
UNEXPECTED EXCEPTION: AttributeError("module 'scipy.sparse' has no attribute 'csgraph'",)
Traceback (most recent call last):

  File "/home/lesteve/miniconda3/envs/test/lib/python3.6/doctest.py", line 1330, in __run
    compileflags, 1), test.globs)

  File "<doctest sklearn.semi_supervised.label_propagation.LabelSpreading[8]>", line 1, in <module>

  File "/home/lesteve/dev/scikit-learn/sklearn/semi_supervised/label_propagation.py", line 229, in fit
    graph_matrix = self._build_graph()

  File "/home/lesteve/dev/scikit-learn/sklearn/semi_supervised/label_propagation.py", line 517, in _build_graph
    laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)

AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

/home/lesteve/dev/scikit-learn/sklearn/semi_supervised/label_propagation.py:483: UnexpectedException
_________________________________________ test_fit_transduction __________________________________________

    def test_fit_transduction():
        samples = [[1., 0.], [0., 2.], [1., 3.]]
        labels = [0, 1, -1]
        for estimator, parameters in ESTIMATORS:
>           clf = estimator(**parameters).fit(samples, labels)

sklearn/semi_supervised/tests/test_label_propagation.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/semi_supervised/label_propagation.py:229: in fit
    graph_matrix = self._build_graph()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LabelSpreading(alpha=0.2, gamma=20, kernel='rbf', max_iter=30, n_jobs=1,
        n_neighbors=7, tol=0.001)

    def _build_graph(self):
        """Graph matrix for Label Spreading computes the graph laplacian"""
        # compute affinity matrix (or gram matrix)
        if self.kernel == 'knn':
            self.nn_fit = None
        n_samples = self.X_.shape[0]
        affinity_matrix = self._get_kernel(self.X_)
>       laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)
E       AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

sklearn/semi_supervised/label_propagation.py:517: AttributeError
___________________________________________ test_distribution ____________________________________________

    def test_distribution():
        samples = [[1., 0.], [0., 1.], [1., 1.]]
        labels = [0, 1, -1]
        for estimator, parameters in ESTIMATORS:
>           clf = estimator(**parameters).fit(samples, labels)

sklearn/semi_supervised/tests/test_label_propagation.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/semi_supervised/label_propagation.py:229: in fit
    graph_matrix = self._build_graph()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LabelSpreading(alpha=0.2, gamma=20, kernel='rbf', max_iter=30, n_jobs=1,
        n_neighbors=7, tol=0.001)

    def _build_graph(self):
        """Graph matrix for Label Spreading computes the graph laplacian"""
        # compute affinity matrix (or gram matrix)
        if self.kernel == 'knn':
            self.nn_fit = None
        n_samples = self.X_.shape[0]
        affinity_matrix = self._get_kernel(self.X_)
>       laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)
E       AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

sklearn/semi_supervised/label_propagation.py:517: AttributeError
______________________________________________ test_predict ______________________________________________

    def test_predict():
        samples = [[1., 0.], [0., 2.], [1., 3.]]
        labels = [0, 1, -1]
        for estimator, parameters in ESTIMATORS:
>           clf = estimator(**parameters).fit(samples, labels)

sklearn/semi_supervised/tests/test_label_propagation.py:56: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/semi_supervised/label_propagation.py:229: in fit
    graph_matrix = self._build_graph()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LabelSpreading(alpha=0.2, gamma=20, kernel='rbf', max_iter=30, n_jobs=1,
        n_neighbors=7, tol=0.001)

    def _build_graph(self):
        """Graph matrix for Label Spreading computes the graph laplacian"""
        # compute affinity matrix (or gram matrix)
        if self.kernel == 'knn':
            self.nn_fit = None
        n_samples = self.X_.shape[0]
        affinity_matrix = self._get_kernel(self.X_)
>       laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)
E       AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

sklearn/semi_supervised/label_propagation.py:517: AttributeError
___________________________________________ test_predict_proba ___________________________________________

    def test_predict_proba():
        samples = [[1., 0.], [0., 1.], [1., 2.5]]
        labels = [0, 1, -1]
        for estimator, parameters in ESTIMATORS:
>           clf = estimator(**parameters).fit(samples, labels)

sklearn/semi_supervised/tests/test_label_propagation.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/semi_supervised/label_propagation.py:229: in fit
    graph_matrix = self._build_graph()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LabelSpreading(alpha=0.2, gamma=20, kernel='rbf', max_iter=30, n_jobs=1,
        n_neighbors=7, tol=0.001)

    def _build_graph(self):
        """Graph matrix for Label Spreading computes the graph laplacian"""
        # compute affinity matrix (or gram matrix)
        if self.kernel == 'knn':
            self.nn_fit = None
        n_samples = self.X_.shape[0]
        affinity_matrix = self._get_kernel(self.X_)
>       laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)
E       AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

sklearn/semi_supervised/label_propagation.py:517: AttributeError
____________________________________ test_label_spreading_closed_form ____________________________________

    def test_label_spreading_closed_form():
        n_classes = 2
        X, y = make_classification(n_classes=n_classes, n_samples=200,
                                   random_state=0)
        y[::3] = -1
>       clf = label_propagation.LabelSpreading().fit(X, y)

sklearn/semi_supervised/tests/test_label_propagation.py:87: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/semi_supervised/label_propagation.py:229: in fit
    graph_matrix = self._build_graph()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LabelSpreading(alpha=0.2, gamma=20, kernel='rbf', max_iter=30, n_jobs=1,
        n_neighbors=7, tol=0.001)

    def _build_graph(self):
        """Graph matrix for Label Spreading computes the graph laplacian"""
        # compute affinity matrix (or gram matrix)
        if self.kernel == 'knn':
            self.nn_fit = None
        n_samples = self.X_.shape[0]
        affinity_matrix = self._get_kernel(self.X_)
>       laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)
E       AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

sklearn/semi_supervised/label_propagation.py:517: AttributeError
____________________________________________ test_valid_alpha ____________________________________________

    def test_valid_alpha():
        n_classes = 2
        X, y = make_classification(n_classes=n_classes, n_samples=200,
                                   random_state=0)
        for alpha in [-0.1, 0, 1, 1.1, None]:
            assert_raises(ValueError,
                          lambda **kwargs:
                          label_propagation.LabelSpreading(**kwargs).fit(X, y),
>                         alpha=alpha)

sklearn/semi_supervised/tests/test_label_propagation.py:137: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/utils/_unittest_backport.py:204: in assertRaises
    return context.handle('assertRaises', args, kwargs)
sklearn/utils/_unittest_backport.py:113: in handle
    callable_obj(*args, **kwargs)
sklearn/semi_supervised/tests/test_label_propagation.py:136: in <lambda>
    label_propagation.LabelSpreading(**kwargs).fit(X, y),
sklearn/semi_supervised/label_propagation.py:229: in fit
    graph_matrix = self._build_graph()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LabelSpreading(alpha=-0.1, gamma=20, kernel='rbf', max_iter=30, n_jobs=1,
        n_neighbors=7, tol=0.001)

    def _build_graph(self):
        """Graph matrix for Label Spreading computes the graph laplacian"""
        # compute affinity matrix (or gram matrix)
        if self.kernel == 'knn':
            self.nn_fit = None
        n_samples = self.X_.shape[0]
        affinity_matrix = self._get_kernel(self.X_)
>       laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)
E       AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

sklearn/semi_supervised/label_propagation.py:517: AttributeError
_________________________________________ test_convergence_speed _________________________________________

    def test_convergence_speed():
        # This is a non-regression test for #5774
        X = np.array([[1., 0.], [0., 1.], [1., 2.5]])
        y = np.array([0, 1, -1])
        mdl = label_propagation.LabelSpreading(kernel='rbf', max_iter=5000)
>       mdl.fit(X, y)

sklearn/semi_supervised/tests/test_label_propagation.py:145: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/semi_supervised/label_propagation.py:229: in fit
    graph_matrix = self._build_graph()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LabelSpreading(alpha=0.2, gamma=20, kernel='rbf', max_iter=5000, n_jobs=1,
        n_neighbors=7, tol=0.001)

    def _build_graph(self):
        """Graph matrix for Label Spreading computes the graph laplacian"""
        # compute affinity matrix (or gram matrix)
        if self.kernel == 'knn':
            self.nn_fit = None
        n_samples = self.X_.shape[0]
        affinity_matrix = self._get_kernel(self.X_)
>       laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)
E       AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

sklearn/semi_supervised/label_propagation.py:517: AttributeError
________________________________________ test_convergence_warning ________________________________________

    def test_convergence_warning():
        # This is a non-regression test for #5774
        X = np.array([[1., 0.], [0., 1.], [1., 2.5]])
        y = np.array([0, 1, -1])
        mdl = label_propagation.LabelSpreading(kernel='rbf', max_iter=1)
>       assert_warns(ConvergenceWarning, mdl.fit, X, y)

sklearn/semi_supervised/tests/test_label_propagation.py:157: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/utils/testing.py:143: in assert_warns
    result = func(*args, **kw)
sklearn/semi_supervised/label_propagation.py:229: in fit
    graph_matrix = self._build_graph()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LabelSpreading(alpha=0.2, gamma=20, kernel='rbf', max_iter=1, n_jobs=1,
        n_neighbors=7, tol=0.001)

    def _build_graph(self):
        """Graph matrix for Label Spreading computes the graph laplacian"""
        # compute affinity matrix (or gram matrix)
        if self.kernel == 'knn':
            self.nn_fit = None
        n_samples = self.X_.shape[0]
        affinity_matrix = self._get_kernel(self.X_)
>       laplacian = sparse.csgraph.laplacian(affinity_matrix, normed=True)
E       AttributeError: module 'scipy.sparse' has no attribute 'csgraph'

sklearn/semi_supervised/label_propagation.py:517: AttributeError
=================================== 9 failed, 4 passed in 0.49 seconds ===================================

This is because scipy.sparse.csgraph is not imported implicitly in scipy 1.0. This snippet for example gives an AttributeError in scipy 1.0 and not in scipy < 1.0:

from scipy import sparse
sparse.csgraph

Now if you are lucky, you import something from scipy.sparse.sgraph before using sparse.csgraph and you don't realise which is probably why the tests are passing when I tried running with scipy 1.0.

This can cause problems in scipy 1.0

amueller · 2017-11-15T17:57:19Z

I'm confused by travis passing. The change looks good but we should make sure that the test cover this.

amueller · 2017-11-15T18:06:46Z

(can confirm locally that running the tests works but running only the semisupervised tests fails ... that seems a bit strange to me)

jnothman · 2017-11-15T22:04:01Z

that's to be expected if one of the tests imports scipy.sparse.csgraph and another only imports scipy.sparse but tries to access csgraph...

jnothman

LGTM. I don't see any others in the repo

lesteve · 2017-11-16T02:40:10Z

that's to be expected if one of the tests imports scipy.sparse.csgraph and another only imports scipy.sparse but tries to access csgraph...

Yep (interesting) side-effect of python import mechanism ...

That also means that LabelSpreading is broken with scipy 1.0. Maybe another motivation for doing a 0.19.2?

jnothman · 2017-11-16T06:33:17Z

maybe.... Though at least there is a workaround of importing csgraph first

lesteve · 2017-11-16T06:46:58Z

Though at least there is a workaround of importing csgraph first

Very good point!

This can cause problems in scipy 1.0

Make scipy.sparse.csgraph imports explicit

d2c0794

This can cause problems in scipy 1.0

lesteve force-pushed the make-csgraph-imports-explicit branch from 1295601 to d2c0794 Compare November 15, 2017 16:51

jnothman approved these changes Nov 15, 2017

View reviewed changes

amueller merged commit f6163e7 into scikit-learn:master Nov 15, 2017

lesteve deleted the make-csgraph-imports-explicit branch November 16, 2017 02:30

jwjohnson314 pushed a commit to jwjohnson314/scikit-learn that referenced this pull request Dec 18, 2017

Make scipy.sparse.csgraph imports explicit (scikit-learn#10145)

ddf9b02

This can cause problems in scipy 1.0

qmick mentioned this pull request Jan 18, 2018

Possible bug in LabelSpreading #10494

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[MRG] Make csgraph imports explicit #10145

[MRG] Make csgraph imports explicit #10145

Uh oh!

lesteve commented Nov 15, 2017 •

edited

Loading

Uh oh!

amueller commented Nov 15, 2017

Uh oh!

amueller commented Nov 15, 2017

Uh oh!

jnothman commented Nov 15, 2017 via email

Uh oh!

jnothman left a comment

Uh oh!

lesteve commented Nov 16, 2017

Uh oh!

jnothman commented Nov 16, 2017 via email

Uh oh!

lesteve commented Nov 16, 2017

Uh oh!

Uh oh!

Uh oh!

[MRG] Make csgraph imports explicit #10145

[MRG] Make csgraph imports explicit #10145

Uh oh!

Conversation

lesteve commented Nov 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amueller commented Nov 15, 2017

Uh oh!

amueller commented Nov 15, 2017

Uh oh!

jnothman commented Nov 15, 2017 via email

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

lesteve commented Nov 16, 2017

Uh oh!

jnothman commented Nov 16, 2017 via email

Uh oh!

lesteve commented Nov 16, 2017

Uh oh!

Uh oh!

lesteve commented Nov 15, 2017 •

edited

Loading