PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices #12327

rth · 2018-10-08T15:20:50Z

When running a the test suite we get a number PendingDeprecationWarning,

PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.
  return matrix(data, dtype=dtype, copy=False)

See for instance https://ci.appveyor.com/project/sklearn-ci/scikit-learn/builds/19341228/job/q6rysq7flma8hi6v Warnings seems to be hidden on Travis so we don't see those there.

This typically happens when we have a sparse matrix, sum along one axis to get a dense one, than do some mathematical operation with this matrix. It should be explicitly converted to an array first.

This has also been reported by @drorata in #11251 (comment). There was previous discussion about this in #11251 but it doesn't look like it's fixed. I also saw this recently when running PyPy CI.

To investigate where this happen it's sufficient to run pytest tests with -Werror::PendingDeprecationWarning to error on those warnings, as they don't seem to be raised with the right stacklevel (see also #7963 (comment) for workarounds)

The text was updated successfully, but these errors were encountered:

jnothman · 2018-10-08T20:30:22Z

We should probably create helper functions to do the reduce and toarray() together.

amueller · 2018-10-11T18:55:41Z

Is this important for 0.20.1? It's not a regression, right?

rth · 2018-10-11T19:02:24Z

It's not a regression, and not a blocker for 0.20.1 (please feel free to untag it), but it still fairly annoying as it means we get PendingDeprecationWarning when using the latest scipy and it's better to fix that as soon as possible.

grassknoted · 2018-10-12T18:42:01Z

Hey, could I take this issue up?

rth · 2018-10-12T21:25:48Z

Yes, thank you @grassknoted

amueller · 2018-10-16T15:52:05Z

@grassknoted are you working on this? otherwise maybe @NicolasHug could pick it up?

NicolasHug · 2018-10-16T16:16:40Z

Ok, I'll pick it up unless @grassknoted updates soon.

grassknoted · 2018-10-17T12:34:40Z

Hey, sorry about the delayed response. I'm working on this.
@amueller would you like me to go about fixing it by reducing and convert it to an array, or is there any other solution you think is better?

rth · 2018-10-18T14:30:45Z

@grassknoted Yes, see #12327 (comment)

grassknoted · 2018-10-19T16:57:52Z

@rth thank you, I'm on it.

grassknoted · 2018-10-21T16:17:25Z

I'm sorry, but I'm new to Scipy, and I've spent the past two days reading the documentation, but I'm unable to find where exactly to make the changes to solve this issue. Any pointers to the files would be really appreciated.
Thank you for being patient with me.

thoo · 2018-10-22T05:46:28Z

@grassknoted Just to remove the warnings, you can just squash with warnings module. I am not sure what @rth has in mind.

NicolasHug · 2018-10-22T13:20:15Z

Just to remove the warnings, you can just squash with warnings module

That would not remove the warnings per se, that would just filter them out and we don't want that.

Like @rth said, when some mathematical computation is being done on a sparse matrix, we need to convert it first to an array in order to avoid the warning.

To be fair the 'easy' tag is a bit misleading because when I looked at it, it wasn't obvious at all where to make the matrix-to-array conversions. The conversions can be done in various places and it takes a good knowledge about the repo to identify where exactly it makes sense to convert an object, and where it makes sense to keep its original type.

rth · 2018-10-22T13:26:45Z

To be fair the 'easy' tag is a bit misleading because when I looked at it, it wasn't obvious at all where to make the matrix-to-array conversions.

Even if you convert warnings to errors as suggested in #12327 (comment) ? That should give a detailed error traceback for each such occurrence, which then need to be fixed (possibly with a helper function). Though I have to admit I have not tried that myself, so it might be more complex that anticipated. Yeah, the easy tag is not always that easy..

NicolasHug · 2018-10-22T15:18:55Z

Even if you convert warnings to errors as suggested in #12327 (comment) ?

Of course it helps a lot but it still takes some judgment to decide what should be changed.

For example the first failing test is in a doctest example:
sklearn/random_projection.py:601: np.mean(transformer.components_ != 0)
Do we want to do the conversion in the example only, or upstream?

Another one is e.g. test_birch_predict which fails on
assert_almost_equal(v_measure_score(nearest_centroid, brc.labels_), 1.0)

because v_measure_score calls homogeneity_completeness_v_measure where you have

    contingency = contingency_matrix(labels_true, labels_pred, sparse=True)
    MI = mutual_info_score(None, None, contingency=contingency)

Should we fix it by passing sparse=False, or by going deeper into mutual_info_score where contingency.sum() happens? In this case we probably want both I guess.

I assume there's going to be a lot of cases like that where it's not easy to decide if it's appropriate to do the conversion.

rth · 2018-10-22T15:24:09Z

I see, this is indeed not that straightforward.

Sorry @grassknoted, let's leave this one to @NicolasHug , as it indeed might be not so great for a first contribution, and we do want to have this fixed for 0.20.1.

There is a number of other open issues marked "help wanted" or "good first issues" where your help would be appreciated.

rth · 2018-10-22T18:23:04Z

@NicolasHug For your first example, basically if X is sparse, np.mean(X) or X.sum() appears to produce this warning, which doesn't make sense as there is no sparse ndarray yet. There the solution could be to filter/ignore the warning. That's what will happen in scipy 1.2 (cf scipy/scipy#9093)

For the second example, I think we still want to keep a sparse contingency matrix, but fix it inside mutual_info_score.

In my initial post, I had more in mind cases where we sum only along one dimension and do indeed end up manipulating a dense np.matrix. For the cases where we sum along all dimensions we would probably have to ignore the warning..

ogrisel · 2018-10-25T07:37:27Z

There is no need to filter warnings triggered by scipy itself since those will already be filtered by the next scipy version: scipy/scipy#8887.

NicolasHug · 2018-10-30T23:28:10Z

Sorry for the lag,
I just ran pytest -Werror::PendingDeprecationWarning in a new virtual env using the master branch of scipy, and got no error.

I guess that means we don't need to do anything?

rth · 2018-10-31T00:01:58Z

Thanks for investigating! Agreed. Closing.

drorata · 2018-10-31T07:25:24Z

Do I understand correctly, with the next release of scipy the problem should be resolved?

rth · 2018-10-31T07:32:03Z

Yes, the next scipy release is expected in a month or so https://mail.python.org/pipermail/scipy-dev/2018-October/023132.html so it shouldn't be too long.

drorata · 2018-12-18T06:31:32Z

SciPy 1.2.0 was just released (https://github.com/scipy/scipy/releases/tag/v1.2.0). What is the best way to use it and avoid the warnings? I tried to explicitly install the updated version (in addition to scikit) but it didn't help.

drorata · 2019-01-30T16:02:53Z

I don't think I installed scipy explicitly rather it is installed as a dependency. I could verify this tomorrow.

NicolasHug · 2019-01-30T21:44:30Z

There is some strange interaction with pytest going on.

scipy/scipy#8887 should have fixed most (if not all) warnings.

from scipy.sparse import csr_matrix
import numpy as np

def test_blah():
    m = np.arange(100).reshape(10, 10)
    m = csr_matrix(m)
    m.sum(axis=1)

test_blah()

Run this with python and get no warning. Run this with pytest, and get the deprecation warning.

I have no idea what's going on...

drorata · 2019-01-31T08:13:04Z

@rth I can confirm that I have scipy@1.2.0

In [1]: import scipy

In [2]: scipy.__version__
Out[2]: '1.2.0'

@NicolasHug I can also confirm that your example behaves the same way on my env.

rth · 2019-01-31T11:01:16Z

There is some strange interaction with pytest going on

Ahh, that would explain it @NicolasHug . It looks like scipy/scipy#8887 also ignores these warnings in pytest sessions -- maybe we should do the same (though it's problematic for downstream projects).

@drorata Thanks for following up on this. Are you seeing those warnings in a pytest session or during interactive use. If the latter, are you modifying warning filters in any way in your application?

drorata · 2019-01-31T11:53:04Z

@rth I face the warnings while running tests.

Isn't it a real problem ignoring these warnings once the deprecation actually happens?

rth · 2019-01-31T14:36:16Z

Isn't it a real problem ignoring these warnings once the deprecation actually happens?

It will probably happen eventually, but for now there are only very preliminary plans to do that (scipy/scipy#8162) and more importantly no viable alternative to scipy.sparse (for scikit-learn at least ) see scipy/scipy#8887 (comment)

So this warning can be safely ignored for now, there is nothing we can do short of silencing it.

@NicolasHug Would you mind opening an issue at scipy with your example? It means that any project that writes unit tests with scipy.sparse will see this warning. It's unclear if the issue is with pytest filtering or the fact that numpy deprecated it despite the existence (and active use of scipy.sparse), but IMO it still should be up to scipy to provide some suggestions of how to deal with such warnings in all downstream projects' tests suites.

NicolasHug · 2019-01-31T15:00:43Z

Opened scipy/scipy#9734

drorata · 2019-02-01T08:15:31Z

In the project where I face the problem, I use pytest and I've added:

filterwarnings=
    ignore:the matrix subclass is not the recommended way*:PendingDeprecationWarning

under the [pytest] section of pytest.ini.

As mentioned in scipy/scipy#9734 (comment) this is sub-optimal but at least now the tests are green :)

rth · 2019-02-02T10:19:01Z

This was fixed for scikit-learn in #13076 however keeping this issue open as this is still an issue for all projects that depend on scikit-learn (when running their test suites), unless they manually filter out the warning. I'm hoping someone comes up with some solution to avoid these manual changes.

This is known scipy/pytest issue, see here: scikit-learn/scikit-learn#12327 (comment)

Gseguelg · 2019-03-05T15:05:06Z

In my case, within my code, changing from ".todense()" to ".toarray()" for sparse matrices removed the warnings. As well as ".sum()" over the some axis on sparse matrix are converted to np.matrix. Here may lie the specific problem. This last one forced me to convert sparse matrices toarray or only by adding use a redefinition with the following function (not fully implemented):

import numpy as np
def sum_rows(A):
    """ Sumation along the rows (add all elements of the columns for each row independently).
        Similar to A.sum(axis=1) but return csr_matrix. Used to avoid pytest deprecation warning.
        A must be csr_matrix to be 'efficient'.
    """
    # about A
    NRows_A, NCols_A = A.shape
    indxrow_A, indxcol_A = A.nonzero()
    # about Aux
    NCols_aux = 1
    NRows_aux = NCols_A
    # drop duplicates of indxcol_A to set ones on Aux (to not multiply > 1)
    row_indx_aux = np.unique(indxcol_A)
    col_indx_aux = [0] * len(row_indx_aux)
    data = [1] * len(indxcol_A)
    Aux = sps__csr_matrix( (data, (row_indx_aux, col_indx_aux)), (NRows_aux, NCols_aux) )
    return A * Aux

If you avoid them, you avoid the warnings with pytest.
Hope my toc helps someone.

cmarmo · 2021-01-29T10:34:27Z

Running

pytest -Werror::PendingDeprecationWarning

with

>>> sklearn.show_versions()  

System:
    python: 3.9.1 (default, Dec  8 2020, 00:00:00)  [GCC 10.2.1 20201125 (Red Hat 10.2.1-9)]
executable: /home/cmarmo/.skldevenv/bin/python
   machine: Linux-5.9.16-200.fc33.x86_64-x86_64-with-glibc2.32

Python dependencies:
          pip: 20.3.3
   setuptools: 49.1.3
      sklearn: 1.0.dev0
        numpy: 1.19.4
        scipy: 1.5.4
       Cython: 0.29.21
       pandas: 1.1.5
   matplotlib: 3.3.3
       joblib: 1.0.0
threadpoolctl: 2.1.0

Built with OpenMP: True

generates 4 failures

test_np_matrix
sklearn/utils/tests/test_validation.py
fixed in TST Ignores PendingDeprecationWarning in tests #19301
test_as_float_array
sklearn/utils/tests/test_validation.py
due to the direct call of np.matrix in the test
fixed in TST Ignores PendingDeprecationWarning in tests #19301
test_vectorizer_inverse_transform[TfidfVectorizer]
in sklearn/feature_extraction/tests/test_text.py
fixed in CLN Fixes PendingDeprecationWarning in CountVectorizer #19299
test_vectorizer_inverse_transform[CountVectorizer]
sklearn/feature_extraction/tests/test_text.py
due to the call of np.asmatrix in
fixed in CLN Fixes PendingDeprecationWarning in CountVectorizer #19299

sklearn/feature_extraction/text.py:1280: in inverse_transform
    X = np.asmatrix(X)

I'm relabeling this as "help wanted", feel free to relabel if my understanding is wrong.

lorentzenchr · 2021-01-29T13:17:07Z

I can confirm all 4 cases with slightly different settings in show_versions().

lorentzenchr · 2021-01-30T12:37:50Z

Fixed by #19299 and #19301.

because matrices have been deprecated in `numpy`. Usage of `scipy.sparse` is causing this issue when conversions to matrices are performed. I changed the: - calls to the method `scipy.sparse.lil.lil_matrix.todense` - to calls to the method `scipy.sparse.lil.lil_matrix.toarray`, to avoid the conversions that raise `PendingDeprecationWarning`s. The warnings are raised by the call to the method `lil_matrix.todense` because this call involves the instantiation of the class `numpy.matrix`, which is deprecated. In contrast, the method `lil_matrix.toarray` creates an instance of `numpy.ndarray`. (In fact, in the module `tulip.abstract.discretization`, wherever the method `lil_matrix.todense` was called, the value that it returned was immediately converted to a `numpy.ndarray`. So calling the method `lil_matrix.toarray` is actually more efficient.) The class `numpy.matrix` is deprecated and will probably be removed in the future. This will happen after arrangements have been made for `scipy.space`. (For these points and more information, read the references listed at the end.) Still, I do not think that continuing to use `scipy.sparse.lil_matrix` until when `numpy` removes matrices is a safe approach. Instead, using `numpy.ndarray` would be safer. Moreover, I do think that there are other data structures that would fit this use case better than sparse matrices. ## Diagnosis I describe below the approach I (eventually) followed to debug this warning, because finding the cause was difficult. The issue is a `PendingDeprecationWarning` issued from `numpy`. This warning is visible in `pytest` runs, but *not* when running the Python test file directly. Moreover, `pytest` reports the warning, and from which test function it originates. The warning itself reads (I have wrapped the lines here): ``` ===================================== warnings summary ====================================== abstract_test.py::transition_directions_test /.../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) ``` So the line in `tulip` that causes the warning cannot be found from the information contained in the warning. The above `PendingDeprecationWarning` was introduced in `numpy` in commit: numpy/numpy@11e9d2a This warning was then ignored in the module `scipy.sparse.__init__`, in `scipy` commit: scipy/scipy@a874bd5 It appears that this configuration of warnings by `scipy` interacts with `pytest` complexly: - when running with `pytest abstract_test.py`, the `PendingDeprecationWarning` is visible, but - when running with `python -X dev -- abstract_test.py`, the `PendingDeprecationWarning` is invisible. This behavior is due to the call: ```python warnings.filterwarnings( 'ignore', message='the matrix subclass is not the recommended way') ``` within `scipy.sparse.__init__.py` (introduced in the `scipy` commit that was mentioned above). Read also: https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning https://www.python.org/dev/peps/pep-0565/ As a result, it is difficult to find the cause within `tulip` of this `PendingDeprecationWarning`. ## Getting a traceback The test function that triggered the `PendingDeprecationWarning` from `numpy` was not failing, so there was no traceback that would indicate which line in `tulip` caused the warning. In addition, there was an earlier warning issued by `matplotlib`. So turning warnings to errors with the argument `-Werror` would cause `pytest` to turn the `matplotlib` warning into an error, and stop before the warning of interest: ```shell pytest -Werror abstract_test.py ``` So first I removed the `matplotlib` warnings (temporarily), by commenting the line `matplotlib.use('Agg')` in the file `abstract_test.py`. This made the warning of interest to become the first warning. I then passed `-Werror` to `pytest`, and this turned the `numpy` warning into an error, which produced the traceback shown below: (The cause of these `matplotlib` warnings (there are two) is in the package `polytope`, and has been addressed there, in commit: tulip-control/polytope@c464818 These changes will become available to `tulip` with the next `polytope` release. Until then, the CI tests of `tulip` will raise these `matplotlib` warnings. These warnings could be explicitly ignored by using `with pytest.warns`.) (The paths to `tulip` in the traceback below lead to the repository's `tulip`, instead of a directory under Python's `site-packages`, because during this phase of debugging I installed `tulip` with `pip install -e .`, to iterate faster while debugging.) ``` ../tulip/abstract/discretization.py:1666: in discretize_switched plot_mode_partitions(merged_abstr, show_ts, only_adjacent) ../tulip/abstract/discretization.py:1673: in plot_mode_partitions axs = swab.plot(show_ts, only_adjacent) ../tulip/abstract/discretization.py:187: in plot ax = ab.plot(show_ts, only_adjacent, color_seed) ../tulip/abstract/discretization.py:403: in plot ax = _plot_abstraction(self, show_ts, only_adjacent, ../tulip/abstract/discretization.py:446: in _plot_abstraction ax = ab.ppp.plot( ../tulip/abstract/prop2partition.py:600: in plot return plot_partition( .../.virtualenvs/.../lib/python3.9/site-packages/polytope/plot.py:90: in plot_partition trans = nx.to_numpy_matrix(trans, nodelist=ppp2trans) .../.virtualenvs/.../lib/python3.9/site-packages/networkx/convert_matrix.py:553: in to_numpy_matrix M = np.asmatrix(A, dtype=dtype) .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: in asmatrix return matrix(data, dtype=dtype, copy=False) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ subtype = <class 'numpy.matrix'> data = array([[1., 0., 0., 0., 0., 0.], [0., 1., 1., 0., 1., 1.], [1., 0., 1., 1., 0., 1.], [1., 0., 0., 1., 0., 0.], [0., 0., 0., 0., 1., 1.], [1., 0., 0., 0., 0., 1.]]) dtype = None, copy = False def __new__(subtype, data, dtype=None, copy=True): > warnings.warn('the matrix subclass is not the recommended way to ' 'represent matrices or deal with linear algebra (see ' 'https://docs.scipy.org/doc/numpy/user/' 'numpy-for-matlab-users.html). ' 'Please adjust your code to use regular ndarray.', PendingDeprecationWarning, stacklevel=2) E PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:116: PendingDeprecationWarning ``` As the traceback shows, the issue is due to a call to the function `networkx.to_numpy_matrix` within the function `polytope.plot.plot_partition`. So avoiding this warning will be possible after the next release of the package `polytope`. (Note that inserting an `assert False` in a suitable line within the function `transition_directions_test` is not an alternative to passing the argument `-Werror`, because the `assert False` will result in a traceback where the `assert` statement appears, instead of a traceback that shows the call stack at the point where the warning was issued.) ## Speeding up debugging using `pytest` Also, since I had to be running `pytest` on the Python file `abstract_test.py`, `pytest` would collect all test functions, and run them. The file `abstract_test.py` happens to contain several slow test functions, so running them all just to observe the results for the one function of interest is not time-efficient. What I did to speed up runs was to rename all `test_*` functions contained in `abstract_test.py`, except for the one function of interest (namely `transition_directions_test`), to identifiers outside the patterns collected by `pytest`. A simpler alternative, for use with larger test files, is to do the opposite: rename only the function of interest to a different pattern, and then change the line `python_functions = ` in the configuration file `pytest.ini`. ## References - numpy/numpy#10142 (DEP: Pending deprecation warning for matrix) - numpy/numpy#10973 (DOC: advise against use of matrix) - scipy/scipy#8887 (MAINT: filter out np.matrix PendingDeprecationWarning's in numpy >=1.15) - scipy/scipy#9734 (PendingDeprecationWarning for np.matrix with pytest) - scikit-learn/scikit-learn#12327 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices) - scikit-learn/scikit-learn#13076 ([MRG] Ignore PendingDepWarnings of matrix subclass with pytest) - cvxpy/cvxpy#567 (NumPy matrix class is pending deprecation and issuing warnings) - cvxpy/cvxpy#637 (RF: Use a 2D np array instead of matrix to represent scalars) - cvxpy/cvxpy#638 (RF: Change np.matrix to np.array in several places) - cvxpy/cvxpy#644 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra) - https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning

because matrices have been deprecated in `numpy`. Usage of `scipy.sparse` is causing this issue when conversions to matrices are performed. I changed the: - calls to the method `scipy.sparse.lil.lil_matrix.todense` - to calls to the method `scipy.sparse.lil.lil_matrix.toarray`, to avoid the conversions that raise `PendingDeprecationWarning`s. The warnings are raised by the call to the method `lil_matrix.todense` because this call involves the instantiation of the class `numpy.matrix`, which is deprecated. In contrast, the method `lil_matrix.toarray` creates an instance of `numpy.ndarray`. (In fact, in the module `tulip.abstract.discretization`, wherever the method `lil_matrix.todense` was called, the value that it returned was immediately converted to a `numpy.ndarray`. So calling the method `lil_matrix.toarray` is actually more efficient.) The class `numpy.matrix` is deprecated and will probably be removed in the future. This will happen after arrangements have been made for `scipy.space`. (For these points and more information, read the references listed at the end.) Still, I do not think that continuing to use `scipy.sparse.lil_matrix` until when `numpy` removes matrices is a safe approach. Instead, using `numpy.ndarray` would be safer. ## Diagnosis I describe below the approach I (eventually) followed to debug this warning, because finding the cause was difficult. The issue is a `PendingDeprecationWarning` issued from `numpy`. This warning is visible in `pytest` runs, but *not* when running the Python test file directly. Moreover, `pytest` reports the warning, and from which test function the warning originates. The warning itself reads (I have wrapped the lines here): ``` ===================================== warnings summary ====================================== abstract_test.py::transition_directions_test /.../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) ``` So the line in `tulip` that causes the warning cannot be found from the information contained in the warning. The above `PendingDeprecationWarning` was introduced in `numpy` in commit: numpy/numpy@11e9d2a This warning was then ignored in the module `scipy.sparse.__init__`, in `scipy` commit: scipy/scipy@a874bd5 It appears that this configuration of warnings by `scipy` interacts with `pytest` complexly: - when running with `pytest abstract_test.py`, the `PendingDeprecationWarning` is visible, but - when running with `python -X dev -- abstract_test.py`, the `PendingDeprecationWarning` is invisible. This behavior is due to the call: ```python warnings.filterwarnings( 'ignore', message='the matrix subclass is not the recommended way') ``` within `scipy.sparse.__init__.py` (introduced in the `scipy` commit that was mentioned above). Read also: https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning https://www.python.org/dev/peps/pep-0565/ As a result, it is difficult to find the cause within `tulip` of this `PendingDeprecationWarning`. ## Getting a traceback The test function that triggered the `PendingDeprecationWarning` from `numpy` was not failing, so there was no traceback that would indicate which line in `tulip` caused the warning. In addition, there was an earlier warning issued by `matplotlib`. So turning warnings to errors with the argument `-Werror` would cause `pytest` to turn the `matplotlib` warning into an error, and stop before the warning of interest: ```shell pytest -Werror abstract_test.py ``` So first I removed the `matplotlib` warnings (temporarily), by commenting the line `matplotlib.use('Agg')` in the file `abstract_test.py`. This made the warning of interest to become the first warning. I then passed `-Werror` to `pytest`, and this turned the `numpy` warning into an error, which produced the traceback shown below: (The cause of these `matplotlib` warnings (there are two) is in the package `polytope`, and has been addressed there, in commit: tulip-control/polytope@c464818 These changes will become available to `tulip` with the next `polytope` release. Until then, the CI tests of `tulip` will raise these `matplotlib` warnings. These warnings could be explicitly ignored by using `with pytest.warns`.) (The paths to `tulip` in the traceback below lead to the repository's `tulip`, instead of a directory under Python's `site-packages`, because during this phase of debugging I installed `tulip` with `pip install -e .`, to iterate faster while debugging.) ``` ../tulip/abstract/discretization.py:1666: in discretize_switched plot_mode_partitions(merged_abstr, show_ts, only_adjacent) ../tulip/abstract/discretization.py:1673: in plot_mode_partitions axs = swab.plot(show_ts, only_adjacent) ../tulip/abstract/discretization.py:187: in plot ax = ab.plot(show_ts, only_adjacent, color_seed) ../tulip/abstract/discretization.py:403: in plot ax = _plot_abstraction(self, show_ts, only_adjacent, ../tulip/abstract/discretization.py:446: in _plot_abstraction ax = ab.ppp.plot( ../tulip/abstract/prop2partition.py:600: in plot return plot_partition( .../.virtualenvs/.../lib/python3.9/site-packages/polytope/plot.py:90: in plot_partition trans = nx.to_numpy_matrix(trans, nodelist=ppp2trans) .../.virtualenvs/.../lib/python3.9/site-packages/networkx/convert_matrix.py:553: in to_numpy_matrix M = np.asmatrix(A, dtype=dtype) .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: in asmatrix return matrix(data, dtype=dtype, copy=False) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ subtype = <class 'numpy.matrix'> data = array([[1., 0., 0., 0., 0., 0.], [0., 1., 1., 0., 1., 1.], [1., 0., 1., 1., 0., 1.], [1., 0., 0., 1., 0., 0.], [0., 0., 0., 0., 1., 1.], [1., 0., 0., 0., 0., 1.]]) dtype = None, copy = False def __new__(subtype, data, dtype=None, copy=True): > warnings.warn('the matrix subclass is not the recommended way to ' 'represent matrices or deal with linear algebra (see ' 'https://docs.scipy.org/doc/numpy/user/' 'numpy-for-matlab-users.html). ' 'Please adjust your code to use regular ndarray.', PendingDeprecationWarning, stacklevel=2) E PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:116: PendingDeprecationWarning ``` As the traceback shows, the issue is due to a call to the function `networkx.to_numpy_matrix` within the function `polytope.plot.plot_partition`. So avoiding this warning will be possible after the next release of the package `polytope`. (Note that inserting an `assert False` in a suitable line within the function `transition_directions_test` is not an alternative to passing the argument `-Werror`, because the `assert False` will result in a traceback where the `assert` statement appears, instead of a traceback that shows the call stack at the point where the warning was issued.) ## Speeding up debugging using `pytest` Also, since I had to be running `pytest` on the Python file `abstract_test.py`, `pytest` would collect all test functions, and run them. The file `abstract_test.py` happens to contain several slow test functions, so running them all just to observe the results for the one function of interest is not time-efficient. Running a single test function using `pytest` is possible by writing: ```shell pytest abstract_test.py::name_of_function ``` ## References - numpy/numpy#10142 (DEP: Pending deprecation warning for matrix) - numpy/numpy#10973 (DOC: advise against use of matrix) - scipy/scipy#8887 (MAINT: filter out np.matrix PendingDeprecationWarning's in numpy >=1.15) - scipy/scipy#9734 (PendingDeprecationWarning for np.matrix with pytest) - scikit-learn/scikit-learn#12327 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices) - scikit-learn/scikit-learn#13076 ([MRG] Ignore PendingDepWarnings of matrix subclass with pytest) - cvxpy/cvxpy#567 (NumPy matrix class is pending deprecation and issuing warnings) - cvxpy/cvxpy#637 (RF: Use a 2D np array instead of matrix to represent scalars) - cvxpy/cvxpy#638 (RF: Change np.matrix to np.array in several places) - cvxpy/cvxpy#644 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra) - https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning

rth added Easy Well-defined and straightforward way to resolve help wanted labels Oct 8, 2018

rth added this to the 0.20.1 milestone Oct 8, 2018

rth mentioned this issue Oct 8, 2018

[MRG+1] Fix scipy-dev-wheels build #11251

Merged

amueller removed the Easy Well-defined and straightforward way to resolve label Oct 24, 2018

amueller mentioned this issue Oct 30, 2018

adjusted_rand_score PendingDeprecationWarning #12493

Closed

rth closed this as completed Oct 31, 2018

NicolasHug mentioned this issue Jan 31, 2019

PendingDeprecationWarning for np.matrix with pytest scipy/scipy#9734

Closed

NicolasHug mentioned this issue Feb 1, 2019

[MRG] Ignore PendingDepWarnings of matrix subclass with pytest #13076

Merged

qinhanmin2014 removed this from the 0.20.3 milestone Feb 2, 2019

qinhanmin2014 removed the help wanted label Feb 2, 2019

tberlok added a commit to tberlok/psecas that referenced this issue Feb 12, 2019

Filter scipy matrix subclass warning when using pytest

f93697c

This is known scipy/pytest issue, see here: scikit-learn/scikit-learn#12327 (comment)

gwarmstrong mentioned this issue Apr 4, 2019

PendingDeprecationWarning for numpy matrix subclass qiime2/q2-feature-table#207

Closed

fbarrios mentioned this issue May 5, 2019

Fix PendingDeprecationWarning summanlp/textrank#57

Closed

cmarmo added help wanted module:test-suite everything related to our tests labels Jan 29, 2021

cmarmo added this to the 1.0 milestone Jan 29, 2021

This was referenced Jan 29, 2021

CLN Fixes PendingDeprecationWarning in CountVectorizer #19299

Merged

TST Ignores PendingDeprecationWarning in tests #19301

Merged

lorentzenchr closed this as completed Jan 30, 2021

PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices #12327

PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices #12327

Comments

rth commented Oct 8, 2018

jnothman commented Oct 8, 2018 via email

amueller commented Oct 11, 2018

rth commented Oct 11, 2018

grassknoted commented Oct 12, 2018

rth commented Oct 12, 2018

amueller commented Oct 16, 2018

NicolasHug commented Oct 16, 2018

grassknoted commented Oct 17, 2018 • edited Loading

rth commented Oct 18, 2018

grassknoted commented Oct 19, 2018 • edited Loading

grassknoted commented Oct 21, 2018

thoo commented Oct 22, 2018

NicolasHug commented Oct 22, 2018

rth commented Oct 22, 2018

NicolasHug commented Oct 22, 2018 • edited Loading

rth commented Oct 22, 2018

rth commented Oct 22, 2018

ogrisel commented Oct 25, 2018

NicolasHug commented Oct 30, 2018 • edited Loading

rth commented Oct 31, 2018

drorata commented Oct 31, 2018

rth commented Oct 31, 2018

drorata commented Dec 18, 2018

drorata commented Jan 30, 2019

NicolasHug commented Jan 30, 2019

drorata commented Jan 31, 2019

rth commented Jan 31, 2019

drorata commented Jan 31, 2019

rth commented Jan 31, 2019 • edited Loading

NicolasHug commented Jan 31, 2019

drorata commented Feb 1, 2019

rth commented Feb 2, 2019

Gseguelg commented Mar 5, 2019 • edited Loading

cmarmo commented Jan 29, 2021 • edited by lorentzenchr Loading

lorentzenchr commented Jan 29, 2021

lorentzenchr commented Jan 30, 2021

grassknoted commented Oct 17, 2018 •

edited

Loading

grassknoted commented Oct 19, 2018 •

edited

Loading

NicolasHug commented Oct 22, 2018 •

edited

Loading

NicolasHug commented Oct 30, 2018 •

edited

Loading

rth commented Jan 31, 2019 •

edited

Loading

Gseguelg commented Mar 5, 2019 •

edited

Loading

cmarmo commented Jan 29, 2021 •

edited by lorentzenchr

Loading