Skip to content

[WIP] loss function name consistency #3556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[WIP] loss function name consistency #3556

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a4b2e6
standardizing names of loss functions
dsullivan7 Aug 13, 2014
79c4979
fixing conflicts
dsullivan7 Aug 13, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions sklearn/ensemble/gradient_boosting.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,7 @@ def _update_terminal_region(self, tree, terminal_regions, leaf, X, y,

LOSS_FUNCTIONS = {'ls': LeastSquaresError,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add "squared".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, actually SGD is using "squared_loss"... The "_loss" suffix is not consistent :-/ I would add the "squared" alias to SGD as well (and make it the default loss for SGDRegressor).

'lad': LeastAbsoluteError,
'absolute': LeastAbsoluteError,
'huber': HuberLossFunction,
'quantile': QuantileLossFunction,
'bdeviance': BinomialDeviance,
Expand Down Expand Up @@ -1239,12 +1240,13 @@ class GradientBoostingRegressor(BaseGradientBoosting, RegressorMixin):

Parameters
----------
loss : {'ls', 'lad', 'huber', 'quantile'}, optional (default='ls')
loss function to be optimized. 'ls' refers to least squares
regression. 'lad' (least absolute deviation) is a highly robust
loss function solely based on order information of the input
loss : {'ls', 'lad', 'absolute', huber', 'quantile'},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

huber' -> 'huber'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you got it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would mention only absolute and not lad in the docstring. Same for squared.

optional (default='ls') loss function to be optimized. 'ls' refers to
least squares regression. 'lad' (least absolute deviation) is a highly
robust loss function solely based on order information of the input
variables. 'huber' is a combination of the two. 'quantile'
allows quantile regression (use `alpha` to specify the quantile).
'absolute' is equivalent to 'lad'.

learning_rate : float, optional (default=0.1)
learning rate shrinks the contribution of each tree by `learning_rate`.
Expand Down Expand Up @@ -1368,7 +1370,7 @@ class GradientBoostingRegressor(BaseGradientBoosting, RegressorMixin):
Elements of Statistical Learning Ed. 2, Springer, 2009.
"""

_SUPPORTED_LOSS = ('ls', 'lad', 'huber', 'quantile')
_SUPPORTED_LOSS = ('ls', 'lad', 'huber', 'quantile', 'absolute')

def __init__(self, loss='ls', learning_rate=0.1, n_estimators=100,
subsample=1.0, min_samples_split=2,
Expand Down
4 changes: 3 additions & 1 deletion sklearn/ensemble/tests/test_gradient_boosting.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,8 @@ def test_loss_function():
GradientBoostingClassifier(loss='ls').fit, X, y)
assert_raises(ValueError,
GradientBoostingClassifier(loss='lad').fit, X, y)
assert_raises(ValueError,
GradientBoostingClassifier(loss='absolute').fit, X, y)
assert_raises(ValueError,
GradientBoostingClassifier(loss='quantile').fit, X, y)
assert_raises(ValueError,
Expand Down Expand Up @@ -162,7 +164,7 @@ def test_classification_synthetic():
def test_boston():
"""Check consistency on dataset boston house prices with least squares
and least absolute deviation. """
for loss in ("ls", "lad", "huber"):
for loss in ("ls", "lad", "absolute", "huber"):
for subsample in (1.0, 0.5):
clf = GradientBoostingRegressor(n_estimators=100, loss=loss,
max_depth=4, subsample=subsample,
Expand Down
16 changes: 13 additions & 3 deletions sklearn/svm/classes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ class LinearSVC(BaseLibLinear, LinearClassifierMixin, _LearntSelectorMixin,
C : float, optional (default=1.0)
Penalty parameter C of the error term.

loss : string, 'l1' or 'l2' (default='l2')
loss : string, {'l1', 'l2', 'hinge', 'squared_hinge', 'lr'} (default='l2')
Specifies the loss function. 'l1' is the hinge loss (standard SVM)
while 'l2' is the squared hinge loss.
while 'l2' is the squared hinge loss. 'hinge' is equivalent to 'l1',
'squared_hinge' is equivalent to 'l2', 'lr' is the logistic
regression

penalty : string, 'l1' or 'l2' (default='l2')
Specifies the norm used in the penalization. The 'l2'
Expand Down Expand Up @@ -135,7 +137,15 @@ class frequencies.

def __init__(self, penalty='l2', loss='l2', dual=True, tol=1e-4, C=1.0,
multi_class='ovr', fit_intercept=True, intercept_scaling=1,
class_weight=None, verbose=0, random_state=None, max_iter=1000):
class_weight=None, verbose=0, random_state=None,
max_iter=1000):
# make aliases for loss functions
loss = loss.lower().strip()
if loss == "hinge":
loss = "l1"
if loss == "squared_hinge":
loss = "l2"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The loss parameter and self.loss must be equal in order for cloning to work. You should do this in the fit method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I'll leave this out then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, because the init method sets self.loss to loss, will this actually be a problem? If someone were to create an object with loss="HINGE", would it be a concern if the cloned (and the initial object) had self.loss=="l1"


super(LinearSVC, self).__init__(
penalty=penalty, loss=loss, dual=dual, tol=tol, C=C,
multi_class=multi_class, fit_intercept=fit_intercept,
Expand Down
8 changes: 5 additions & 3 deletions sklearn/svm/tests/test_svm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -433,13 +433,15 @@ def test_linearsvc_parameters():
"""
# generate list of possible parameter combinations
params = [(dual, loss, penalty) for dual in [True, False]
for loss in ['l1', 'l2', 'lr'] for penalty in ['l1', 'l2']]
for loss in ['l1', 'l2', 'HINGE', ' squared_hinge', 'lr']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HINGE => hinge

for penalty in ['l1', 'l2']]

for dual, loss, penalty in params:
if loss == 'l1' and penalty == 'l1':
if ((loss == 'l1' or loss == 'HINGE') and penalty == 'l1'):
assert_raises(ValueError, svm.LinearSVC, penalty=penalty,
loss=loss, dual=dual)
elif loss == 'l1' and penalty == 'l2' and not dual:
elif ((loss == 'l1' or loss == 'HINGE') and penalty == 'l2'
and not dual):
assert_raises(ValueError, svm.LinearSVC, penalty=penalty,
loss=loss, dual=dual)
elif penalty == 'l1' and dual:
Expand Down