Skip to content

Test case failed (Nondeterministic): test_multinomial_logistic_regression_string_inputs() #7966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
chenhe95 opened this issue Dec 2, 2016 · 8 comments
Closed

Test case failed (Nondeterministic): test_multinomial_logistic_regression_string_inputs() #7966

chenhe95 opened this issue Dec 2, 2016 · 8 comments
Labels
Easy Well-defined and straightforward way to resolve
Milestone
0.19

Comments

@chenhe95
Copy link
Contributor

chenhe95 commented Dec 2, 2016 •
edited
Loading

Description

#7962 Reported a failure in AppVeyor, but I ran the test case on my own computer for the latest master (December 2, 2016) of scikit-learn, and the test case also fails.

The test case fails for
sklearn.linear_model.tests.test_logistic.test_multinomial_logistic_regression_string_inputs()

Here is the output

>>> t.test_multinomial_logistic_regression_string_inputs()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sklearn/linear_model/tests/test_logistic.py", line 437, in test_multinomial_logistic_regression_string_inputs
    assert_equal(sorted(np.unique(lr_cv_str.predict(X_ref))), ['bar', 'baz'])
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 744, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 726, in assertSequenceEqual
    self.fail(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: ['baz'] != ['bar', 'baz']

First differing element 0:
baz
bar

Second list contains 1 additional elements.
First extra element 1:
baz

- ['baz']
+ ['bar', 'baz']

Steps/Code to Reproduce

Download and build December 2, 2016 master version of scikit-learn.
Run the test case, multiple times (like 200?)
The chance of failing is like 1/20

>>> import sklearn.linear_model.tests.test_logistic as t
>>> t.test_multinomial_logistic_regression_string_inputs()
>>> for i in xrange(200):
...     t.test_multinomial_logistic_regression_string_inputs()
...

Expected Results

The test case does not fail.

Actual Results

The test case fails. (See output log)

Versions

Darwin-15.6.0-x86_64-i386-64bit
('Python', '2.7.10 (default, Oct 23 2015, 19:19:21) \n[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]')
('NumPy', '1.11.1')
('SciPy', '0.13.0b1')
('Scikit-Learn', '0.19.dev0')
@chenhe95 chenhe95 changed the title Test case failed: test_multinomial_logistic_regression_string_inputs() Test case failed (Nondeterministic): test_multinomial_logistic_regression_string_inputs() Dec 2, 2016
@amueller
Copy link
Member

amueller commented Dec 2, 2016

can you check if the coefficients of lr_cv_str change for multiple runs of l-bfgs? The random_state is not used anywhere in the lbfgs code path as far as I can see.

@amueller
Copy link
Member

amueller commented Dec 2, 2016

lol make_classification is run without random_state.

@amueller amueller added Easy Well-defined and straightforward way to resolve Need Contributor labels Dec 2, 2016
@amueller
Copy link
Member

amueller commented Dec 2, 2016

@chenhe95 can you check that fixing the random state in line 406 fixes the issue?

@chenhe95
Copy link
Contributor Author

chenhe95 commented Dec 2, 2016 •
edited
Loading

Sure thing. Will do after done with class today.

@chenhe95
Copy link
Contributor Author

chenhe95 commented Dec 2, 2016 •
edited
Loading

I think setting the random state fixed it. Ran for 1000 iterations with no error. Setting the random state makes sense too.

@amueller
Copy link
Member

amueller commented Dec 2, 2016

yeah all random states should always be set in the test. Do you wanna sent a PR?

@ivannz ivannz mentioned this issue Dec 2, 2016
Open
@chenhe95 chenhe95 mentioned this issue Dec 2, 2016
Merged
@chenhe95
Copy link
Contributor Author

chenhe95 commented Dec 2, 2016

@amueller I made the PR #7968

@raghavrv raghavrv added this to the 0.19 milestone Dec 2, 2016
@agramfort
Copy link
Member

closed by #7968

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Easy Well-defined and straightforward way to resolve
Projects
None yet
Milestone
0.19
Development

No branches or pull requests
4 participants
@agramfort @amueller @raghavrv @chenhe95