Skip to content

[MRG] Change default solver in LogisticRegression #11476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 20 commits into from
Closed

[MRG] Change default solver in LogisticRegression #11476

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
8f708da
Change default solver in LogisticRegression
TomDLT Jul 6, 2018
83a5dec
simpler auto solver
TomDLT Jul 27, 2018
25f343d
Merge branch 'master' into logistic_default
TomDLT Jul 27, 2018
679224b
fix tests
TomDLT Jul 27, 2018
a9219e8
fix more tests
TomDLT Jul 28, 2018
284ecaa
fix doctests
TomDLT Jul 29, 2018
cc53212
remove 'auto' solver
TomDLT Jul 29, 2018
57b8364
add whats_new entry
TomDLT Jul 29, 2018
1e21983
fix doctests
TomDLT Jul 29, 2018
85f0ecc
cleaning
TomDLT Jul 30, 2018
d06bad1
only warn for multi_class default if the problem is not binary
TomDLT Jul 31, 2018
636e1c0
Merge branch 'master' into logistic_default
TomDLT Aug 9, 2018
94e7927
minor change
TomDLT Aug 9, 2018
7342030
FIX doctests
TomDLT Aug 9, 2018
0ccf0b9
FIX doctest...
TomDLT Aug 10, 2018
115d34d
Revert "only warn for multi_class default if the problem is not binary"
TomDLT Aug 10, 2018
91be8bf
This is an API change, not a feature
jnothman Aug 13, 2018
f7cd2fe
Merge branch 'master' into logistic_default
rth Aug 16, 2018
bf3a7fd
Decrease numerical precision in LogisticRegression doctest
rth Aug 17, 2018
ec86d3b
More review comments
rth Aug 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions doc/modules/ensemble.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -975,7 +975,8 @@ The following example shows how to fit the majority rule classifier::
>>> iris = datasets.load_iris()
>>> X, y = iris.data[:, 1:3], iris.target

>>> clf1 = LogisticRegression(random_state=1)
>>> clf1 = LogisticRegression(solver='lbfgs', multi_class='multinomial',
... random_state=1)
>>> clf2 = RandomForestClassifier(n_estimators=50, random_state=1)
>>> clf3 = GaussianNB()

Expand All @@ -984,10 +985,10 @@ The following example shows how to fit the majority rule classifier::
>>> for clf, label in zip([clf1, clf2, clf3, eclf], ['Logistic Regression', 'Random Forest', 'naive Bayes', 'Ensemble']):
... scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')
... print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))
Accuracy: 0.90 (+/- 0.05) [Logistic Regression]
Accuracy: 0.95 (+/- 0.04) [Logistic Regression]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lol. Now the example is not very interesting any more.

Accuracy: 0.94 (+/- 0.04) [Random Forest]
Accuracy: 0.91 (+/- 0.04) [naive Bayes]
Accuracy: 0.95 (+/- 0.05) [Ensemble]
Accuracy: 0.95 (+/- 0.04) [Ensemble]


Weighted Average Probabilities (Soft Voting)
Expand Down Expand Up @@ -1060,7 +1061,8 @@ The `VotingClassifier` can also be used together with `GridSearch` in order
to tune the hyperparameters of the individual estimators::

>>> from sklearn.model_selection import GridSearchCV
>>> clf1 = LogisticRegression(random_state=1)
>>> clf1 = LogisticRegression(solver='lbfgs', multi_class='multinomial',
... random_state=1)
>>> clf2 = RandomForestClassifier(random_state=1)
>>> clf3 = GaussianNB()
>>> eclf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2), ('gnb', clf3)], voting='soft')
Expand Down
23 changes: 14 additions & 9 deletions doc/modules/linear_model.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -775,15 +775,20 @@ The "saga" solver [7]_ is a variant of "sag" that also supports the
non-smooth `penalty="l1"` option. This is therefore the solver of choice
for sparse multinomial logistic regression.

In a nutshell, one may choose the solver with the following rules:

================================= =====================================
Case Solver
================================= =====================================
L1 penalty "liblinear" or "saga"
Multinomial loss "lbfgs", "sag", "saga" or "newton-cg"
Very Large dataset (`n_samples`) "sag" or "saga"
================================= =====================================
In a nutshell, the following table summarizes the solvers characteristics:

============================ =========== ======= =========== ===== ======
solver 'liblinear' 'lbfgs' 'newton-cg' 'sag' 'saga'
============================ =========== ======= =========== ===== ======
Multinomial + L2 penalty no yes yes yes yes
OVR + L2 penalty yes yes yes yes yes
Multinomial + L1 penalty no no no no yes
OVR + L1 penalty yes no no no yes
============================ =========== ======= =========== ===== ======
Penalize the intercept (bad) yes no no no no
Faster for large datasets no no no yes yes
Robust to unscaled datasets yes yes yes no no
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table is great!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah really good!

============================ =========== ======= =========== ===== ======

The "saga" solver is often the best choice. The "liblinear" solver is
used by default for historical reasons.
Expand Down
11 changes: 6 additions & 5 deletions doc/tutorial/statistical_inference/supervised_learning.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -374,12 +374,13 @@ function or **logistic** function:

::

>>> logistic = linear_model.LogisticRegression(C=1e5)
>>> logistic.fit(iris_X_train, iris_y_train)
>>> log = linear_model.LogisticRegression(solver='lbfgs', C=1e5,
... multi_class='multinomial')
>>> log.fit(iris_X_train, iris_y_train) # doctest: +NORMALIZE_WHITESPACE
LogisticRegression(C=100000.0, class_weight=None, dual=False,
fit_intercept=True, intercept_scaling=1, max_iter=100,
multi_class='ovr', n_jobs=None, penalty='l2', random_state=None,
solver='liblinear', tol=0.0001, verbose=0, warm_start=False)
fit_intercept=True, intercept_scaling=1, max_iter=100,
multi_class='multinomial', n_jobs=None, penalty='l2', random_state=None,
solver='lbfgs', tol=0.0001, verbose=0, warm_start=False)

This is known as :class:`LogisticRegression`.

Expand Down
7 changes: 7 additions & 0 deletions doc/whats_new/v0.20.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ Support for Python 3.3 has been officially dropped.
by `Andreas Müller`_ and :user:`Guillaume Lemaitre <glemaitre>`.



:mod:`sklearn.covariance`
.........................

Expand Down Expand Up @@ -504,6 +505,12 @@ Support for Python 3.3 has been officially dropped.
ValueError.
:issue:`11327` by :user:`Karan Dhingra <kdhingra307>` and `Joel Nothman`_.

- |API| The default values of the ``solver`` and ``multi_class`` parameters of
:class:`linear_model.LogisticRegression` will change respectively from
``'liblinear'`` and ``'ovr'`` in version 0.20 to ``'lbfgs'`` and
``'multinomial'`` in version 0.22. A FutureWarning is raised when the default
values are used. :issue:`11476` by `Tom Dupre la Tour`_.

- |API| Deprecate ``positive=True`` option in :class:`linear_model.Lars` as
the underlying implementation is broken. Use :class:`linear_model.Lasso`
instead. :issue:`9837` by `Alexandre Gramfort`_.
Expand Down
6 changes: 6 additions & 0 deletions sklearn/ensemble/tests/test_bagging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,8 @@ def test_bootstrap_features():
assert_greater(boston.data.shape[1], np.unique(features).shape[0])


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
def test_probability():
# Predict probabilities.
rng = check_random_state(0)
Expand Down Expand Up @@ -712,6 +714,8 @@ def test_oob_score_consistency():
assert_equal(bagging.fit(X, y).oob_score_, bagging.fit(X, y).oob_score_)


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
def test_estimators_samples():
# Check that format of estimators_samples_ is correct and that results
# generated at fit time can be identically reproduced at a later time
Expand Down Expand Up @@ -748,6 +752,8 @@ def test_estimators_samples():
assert_array_almost_equal(orig_coefs, new_coefs)


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
def test_estimators_samples_deterministic():
# This test is a regression test to check that with a random step
# (e.g. SparseRandomProjection) and a given random state, the results
Expand Down
31 changes: 30 additions & 1 deletion sklearn/ensemble/tests/test_voting_classifier.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@
X, y = iris.data[:, 1:3], iris.target


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
def test_estimator_init():
eclf = VotingClassifier(estimators=[])
msg = ('Invalid `estimators` attribute, `estimators` should be'
Expand Down Expand Up @@ -59,6 +61,8 @@ def test_estimator_init():
assert_raise_message(ValueError, msg, eclf.fit, X, y)


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
def test_predictproba_hardvoting():
eclf = VotingClassifier(estimators=[('lr1', LogisticRegression()),
('lr2', LogisticRegression())],
Expand All @@ -67,6 +71,8 @@ def test_predictproba_hardvoting():
assert_raise_message(AttributeError, msg, eclf.predict_proba, X)


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
def test_notfitted():
eclf = VotingClassifier(estimators=[('lr1', LogisticRegression()),
('lr2', LogisticRegression())],
Expand All @@ -76,6 +82,8 @@ def test_notfitted():
assert_raise_message(NotFittedError, msg, eclf.predict_proba, X)


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_majority_label_iris():
"""Check classification by majority label on dataset iris."""
Expand All @@ -92,7 +100,8 @@ def test_majority_label_iris():
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_tie_situation():
"""Check voting classifier selects smaller class label in tie situation."""
clf1 = LogisticRegression(random_state=123)
clf1 = LogisticRegression(random_state=123, multi_class='ovr',
solver='liblinear')
clf2 = RandomForestClassifier(random_state=123)
eclf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2)],
voting='hard')
Expand All @@ -101,6 +110,8 @@ def test_tie_situation():
assert_equal(eclf.fit(X, y).predict(X)[73], 1)


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_weights_iris():
"""Check classification by average probabilities on dataset iris."""
Expand All @@ -115,6 +126,8 @@ def test_weights_iris():
assert_almost_equal(scores.mean(), 0.93, decimal=2)


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_predict_on_toy_problem():
"""Manually check predicted class labels for toy dataset."""
Expand Down Expand Up @@ -148,6 +161,8 @@ def test_predict_on_toy_problem():
assert_equal(all(eclf.fit(X, y).predict(X)), all([1, 1, 1, 2, 2, 2]))


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_predict_proba_on_toy_problem():
"""Calculate predicted probabilities on toy dataset."""
Expand Down Expand Up @@ -216,6 +231,8 @@ def test_multilabel():
return


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_gridsearch():
"""Check GridSearch support."""
Expand All @@ -234,6 +251,8 @@ def test_gridsearch():
grid.fit(iris.data, iris.target)


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_parallel_fit():
"""Check parallel backend of VotingClassifier on toy dataset."""
Expand All @@ -256,6 +275,8 @@ def test_parallel_fit():
assert_array_almost_equal(eclf1.predict_proba(X), eclf2.predict_proba(X))


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_sample_weight():
"""Tests sample_weight parameter of VotingClassifier"""
Expand Down Expand Up @@ -300,6 +321,8 @@ def fit(self, X, y, *args, **sample_weight):
eclf.fit(X, y, sample_weight=np.ones((len(y),)))


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_set_params():
"""set_params should be able to set estimators"""
Expand Down Expand Up @@ -335,6 +358,8 @@ def test_set_params():
eclf1.get_params()["lr"].get_params()['C'])


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_set_estimator_none():
"""VotingClassifier set_params should be able to set estimators as None"""
Expand Down Expand Up @@ -390,6 +415,8 @@ def test_set_estimator_none():
assert_array_equal(eclf2.transform(X1), np.array([[0], [1]]))


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_estimator_weights_format():
# Test estimator weights inputs as list and array
Expand All @@ -408,6 +435,8 @@ def test_estimator_weights_format():
assert_array_almost_equal(eclf1.predict_proba(X), eclf2.predict_proba(X))


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
def test_transform():
"""Check transform method of VotingClassifier on toy dataset."""
Expand Down
3 changes: 2 additions & 1 deletion sklearn/ensemble/voting_classifier.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,8 @@ class VotingClassifier(_BaseComposition, ClassifierMixin, TransformerMixin):
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.naive_bayes import GaussianNB
>>> from sklearn.ensemble import RandomForestClassifier, VotingClassifier
>>> clf1 = LogisticRegression(random_state=1)
>>> clf1 = LogisticRegression(solver='lbfgs', multi_class='multinomial',
... random_state=1)
>>> clf2 = RandomForestClassifier(n_estimators=50, random_state=1)
>>> clf3 = GaussianNB()
>>> X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
Expand Down
5 changes: 4 additions & 1 deletion sklearn/feature_selection/tests/test_from_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
from sklearn.utils.testing import assert_greater
from sklearn.utils.testing import assert_array_almost_equal
from sklearn.utils.testing import assert_array_equal
from sklearn.utils.testing import assert_almost_equal
from sklearn.utils.testing import assert_allclose
from sklearn.utils.testing import assert_raises
from sklearn.utils.testing import skip_if_32bit
Expand Down Expand Up @@ -178,6 +177,8 @@ def test_feature_importances():
assert_array_almost_equal(X_new, X[:, feature_mask])


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
def test_sample_weight():
# Ensure sample weights are passed to underlying estimator
X, y = datasets.make_classification(
Expand Down Expand Up @@ -214,6 +215,8 @@ def test_coef_default_threshold():
assert_array_almost_equal(X_new, X[:, mask])


@pytest.mark.filterwarnings('ignore: Default solver will be changed') # 0.22
@pytest.mark.filterwarnings('ignore: Default multi_class will') # 0.22
@skip_if_32bit
def test_2d_coef():
X, y = datasets.make_classification(
Expand Down
Loading