Indeed, I think that we have an issue there. We resampling during the parallel fit (in your example, it should only correspond to a data shuffling) and then generate the bootstrap. However, the way that OOB score are currently computed, does not take into account the sampling. It means that we have samples in OOB which are actually used by the trees, explaining the over-optimistic score obtained.

We should repimplement (set_oob_score) to take into account the sampling on X>

Thanks @glemaitre for the super fast reply. However, reinstalling (via conda) did not change anything for me and installing from github broke for me:

ModuleNotFoundError Traceback (most recent call last)
in
4 import scipy; print("SciPy", scipy.version)
5 import sklearn; print("Scikit-Learn", sklearn.version)
----> 6 import imblearn; print("Imbalanced-Learn", imblearn.version)

~\Anaconda3\envs\imblearn\lib\site-packages\imblearn_init_.py in
32 Module which allowing to create pipeline with scikit-learn estimators.
33 """
---> 34 from . import combine
35 from . import ensemble
36 from . import exceptions

~\Anaconda3\envs\imblearn\lib\site-packages\imblearn\combine_init_.py in
3 """
4
----> 5 from ._smote_enn import SMOTEENN
6 from ._smote_tomek import SMOTETomek
7

~\Anaconda3\envs\imblearn\lib\site-packages\imblearn\combine_smote_enn.py in
8 from sklearn.utils import check_X_y
9
---> 10 from ..base import BaseSampler
11 from ..over_sampling import SMOTE
12 from ..over_sampling.base import BaseOverSampler

~\Anaconda3\envs\imblearn\lib\site-packages\imblearn\base.py in
14 from sklearn.utils.multiclass import check_classification_targets
15
---> 16 from .utils import check_sampling_strategy, check_target_type
17
18

~\Anaconda3\envs\imblearn\lib\site-packages\imblearn\utils_init_.py in
5 from ._docstring import Substitution
6
----> 7 from ._validation import check_neighbors_object
8 from ._validation import check_target_type
9 from ._validation import check_sampling_strategy

~\Anaconda3\envs\imblearn\lib\site-packages\imblearn\utils_validation.py in
11
12 from sklearn.base import clone
---> 13 from sklearn.neighbors._base import KNeighborsMixin
14 from sklearn.neighbors import NearestNeighbors
15 from sklearn.utils.multiclass import type_of_target
ModuleNotFoundError: No module named 'sklearn.neighbors._base'

When will this fix be available via conda-forge or PyPi?

The package has been updated on PyPI, 20 minutes ago. Be aware that running imbalanced-learn 0.6 will require scikit-learn 0.22 (all available on PyPI). Imbalanced-learn will be shortly available in conda-forge (probably at the end of the day).

I think that scikit-learn is still not available on the conda channel (but soonish as well).

Works. Thanks a lot!

Hi @glemaitre after some investigation it appears that the bug is not completely gone. The oob score is what it should be in the balanced case (example in my initial post) but in an unbalanced case it is still far too high.
`
import numpy as np
from imblearn import ensemble
from sklearn import metrics

X = np.arange(1000).reshape(-1, 1)
y = np.random.binomial(1, 0.1, size=1000)

rf = ensemble.BalancedRandomForestClassifier(oob_score=True, n_estimators=1000)
rf.fit(X, y)

print(rf.oob_score_)

conf_mat = metrics.confusion_matrix(y_true=y, y_pred=rf.oob_decision_function_.argmax(axis=1))

print(conf_mat)
`
gives me
0.972
[[880 26]
[ 2 92]]

The confusion matrix shows that the classifier classifies the minority class perfectly while being ok on the negative class. This should not be possible in this case.

My versions are now
Windows-10-10.0.18362-SP0
Python 3.8.0 | packaged by conda-forge | (default, Nov 22 2019, 19:04:36) [MSC v.1916 64 bit (AMD64)]
NumPy 1.17.3
SciPy 1.3.2
Scikit-Learn 0.22
Imbalanced-Learn 0.6.0

OK found it, some bad interaction with the new parameters max_samples from the trees.

The fix is there: #661