Skip to content

Run itrees in parallel during prediction. #14001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

Run itrees in parallel during prediction. #14001

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c8eab23
Run itrees in parallel during prediction.
May 31, 2019
be93e30
Trees parallelized in chunks during prediction.
Jun 28, 2019
07c328b
Changed the name of the variables in inner get_depths function in ifo…
sergiormpereira Jul 1, 2019
1afb725
Merge pull request #1 from TechhubLisbon/iforest-parallel-predict-chunks
sergiormpereira Jul 1, 2019
16f1e3f
Turned flag of delayed function check_pickle to False, when called in…
sergiormpereira Jul 2, 2019
35c9951
Added condition to check pickle depending on the version of joblib.
sergiormpereira Jul 2, 2019
9c9f43f
Changed loop to np.sum in _compute_score_samples of IForest.
sergiormpereira Jul 16, 2019
fbdbcce
Included test of score samples in parallel in Isolation Forest
sergiormpereira Jul 16, 2019
1be8752
Added PR 14001 to whats_new/v0.22.rst. IForest now runs parallel jobs…
sergiormpereira Jul 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/whats_new/v0.22.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ Changelog
validation data separately to avoid any data leak. :pr:`13933` by
`NicolasHug`_.

- |Fix| :class:`ensemble.IsolationForest` now runs parallel jobs
during :term:`predict`. :pr:`14001` by :user:`Sérgio Pereira
<sergiormpereira>`.

:mod:`sklearn.linear_model`
..................

Expand Down
61 changes: 42 additions & 19 deletions sklearn/ensemble/iforest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@
# Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
# License: BSD 3 clause

from distutils.version import LooseVersion

import joblib
from joblib import Parallel, delayed
import numbers
import numpy as np
from scipy.sparse import issparse
Expand All @@ -20,6 +23,7 @@
from ..base import OutlierMixin

from .bagging import BaseBagging
from .base import _partition_estimators

__all__ = ["IsolationForest"]

Expand Down Expand Up @@ -414,37 +418,56 @@ def _compute_chunked_score_samples(self, X):
return scores

def _compute_score_samples(self, X, subsample_features):
"""Compute the score of each samples in X going through the extra trees.
"""Compute the score of each samples in X going through the extra
trees.

Parameters
----------
X : array-like or sparse matrix

subsample_features : bool,
whether features should be subsampled

Returns
-------
ndarray
The anomaly scores for each sample
"""
n_samples = X.shape[0]
def get_depths(_X, trees, trees_features, _subsample_features):
n = _X.shape[0]
batch_depths = np.zeros(n, order="f")

for tree, features in zip(trees, trees_features):
X_subset = _X[:, features] if _subsample_features else _X

leaves_index = tree.apply(X_subset)
node_indicator = tree.decision_path(X_subset)
n_samples_leaf = tree.tree_.n_node_samples[leaves_index]

batch_depths += np.ravel(node_indicator.sum(axis=1)) \
+ _average_path_length(n_samples_leaf) - 1.0

return batch_depths

n_jobs, n_estimators, starts = _partition_estimators(
self.n_estimators, self.n_jobs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this issue again, I think we can do this:

Suggested change
self.n_estimators, self.n_jobs)
self.n_estimators, None)

which allows joblib.parallel_backend to control n_jobs. At a higher level, we can use parallel_backend to control n_jobs:

with parallel_backend("loky", n_jobs=6):
    iso.score_samples(X)

Details: Seeing that _partition_estimators uses effective_n_jobs:

scikit-learn/sklearn/ensemble/_base.py

Line 209 in 8d295fb

n_jobs = min(effective_n_jobs(n_jobs), n_estimators)

effective_n_jobs queries configuration as follows:

with parallel_backend("loky", n_jobs=4):
    print(effective_n_jobs(None))
# 4

# default is 1
print(effective_n_jobs(None))
# 1


depths = np.zeros(n_samples, order="f")
old_joblib = LooseVersion(joblib.__version__) < LooseVersion('0.12')
check_pickle = False if old_joblib else None

for tree, features in zip(self.estimators_, self.estimators_features_):
X_subset = X[:, features] if subsample_features else X
par_exec = Parallel(n_jobs=n_jobs, **self._parallel_args())
par_results = par_exec(
delayed(get_depths, check_pickle=check_pickle)(
_X=X, trees=self.estimators_[starts[i]: starts[i + 1]],
trees_features=self.estimators_features_[
starts[i]: starts[i + 1]],
_subsample_features=subsample_features)
for i in range(n_jobs))

leaves_index = tree.apply(X_subset)
node_indicator = tree.decision_path(X_subset)
n_samples_leaf = tree.tree_.n_node_samples[leaves_index]
depths = np.sum(par_results, axis=0)

depths += (
np.ravel(node_indicator.sum(axis=1))
+ _average_path_length(n_samples_leaf)
- 1.0
)
scores = 2 ** (-depths / (len(self.estimators_)
* _average_path_length([self.max_samples_])))

scores = 2 ** (
-depths
/ (len(self.estimators_)
* _average_path_length([self.max_samples_]))
)
return scores


Expand Down
5 changes: 5 additions & 0 deletions sklearn/ensemble/tests/test_iforest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,15 +169,20 @@ def test_iforest_parallel_regression():

ensemble.set_params(n_jobs=1)
y1 = ensemble.predict(X_test)
scores1 = ensemble.score_samples(X_test)
ensemble.set_params(n_jobs=2)
y2 = ensemble.predict(X_test)
scores2 = ensemble.score_samples(X_test)
assert_array_almost_equal(y1, y2)
assert_array_almost_equal(scores1, scores2)

ensemble = IsolationForest(n_jobs=1,
random_state=0).fit(X_train)

y3 = ensemble.predict(X_test)
scores3 = ensemble.score_samples(X_test)
assert_array_almost_equal(y1, y3)
assert_array_almost_equal(scores1, scores3)


def test_iforest_performance():
Expand Down