Yes this has not gathered a lot of citations in general but a typical amount for a good paper in this subfield.

There are other solutions to the issue of imbalanced data. The "class_weight" method which is already implemented is a traditional but inferior solution. Other methods which receive comparable results are mostly ensemble methods which rely on balancing at the time of bagging. Good examples of this are Random Balance

https://www.researchgate.net/publication/276152298_Random_Balance_Ensembles_of_variable_priors_classifiers_for_imbalanced_data

or RUSBoost
http://ieeexplore.ieee.org/document/5299216/

Similar methods are implemented in
http://contrib.scikit-learn.org/imbalanced-learn/stable/api.html#module-imblearn.ensemble

You could argue that split criterion based solutions to imbalanced data should be in that package. I do not know where to draw the line since this minor feature addition has been shown to outperform the bulk of the methods in that package. There is one other comparable method to the Hellinger method
https://www3.nd.edu/~nchawla/papers/SDM10.pdf

It is another splitting criterion method called Class Confidence Proportion which does an equivalent yet different thing. I would be happy with this instead of the Hellinger method.

I am not sure using citations as a metric for the usefulness of a feature is best. This is a very specific problem that occurs quite frequently in practice. Many users of scikit-learn are happy to use traditional methods or the ensemble methods in the imbalanced-learn package. It is easy to see there is general interest in solving this issue in this way

https://www.reddit.com/r/MachineLearning/comments/2m72hv/hellinger_distance_decision_trees/

In any case, please leave the ticket open until some criterion based solution is implemented for the community even if this ticket needs to be passed to imbalanced-learn. Thanks!

I think that we could integrate it in imbalanced-learn if it shows some improvement.

The criterion in the tree in scikit-learn are classes; therefore I think that we could implement our own criterion on our side. We will just need scikit-learn to allow us for duck-typing.

@jnothman Do you think this is something reasonable?

That is a great solution. It would allow to have all the imbalanced classification tools in one package. It would seem the change to Scikit-learn is relatively small. Could a ticket for this change be made to supplant this one and this be passed to imbalanced learn? What is the release schedule like? Would getting this into 0.19.1 be possible?

@DrEhrfurchtgebietend @glemaitre did this get implemented? I can't find it in imbalanced-learn. I'm trying to subclass (Classification)Criterion myself and am looking for an example, so if you know of anything like that, that would be very much appreciated.

I've implemented Hellinger Distance as a split criterion for sklearn DecisionTreeClassifier and RandomForestClassifier.
It performs great in my use cases of imbalanced data classification, beats RandomForestClassifier with gini and XGBClassifier.
You are welcome to check it out on https://github.com/EvgeniDubov/hellinger-distance-criterion

This is great @EvgeniDubov . Unless I misunderstood your documentation. The necessary change to scikit_learn is already implemented. I am running 0.19.1 which does not have it yet, is there an idea about when it will be added to the production version?

The criterion is then an independent package to get the criteria. It would make a lot of sense for this to be added to imbalanced-learn with the rest of the imbalanced data handling methods. @glemaitre do you have concerns other than dependencies on scikit_learn?

Thanks @DrEhrfurchtgebietend .
Hellinger Distance helps in imbalanced data classification but also works with balanced data sets.
Looks like imbalanced-learn is mainly focused on data sampling.
It might be a good idea to open a new sklearn-contrib repository where current and future Cython implementations of both ClassificationCriterion and RegressionCriterion will be placed.
I'm willing to open such a project and submit it for your review and feedbacks. What do you say @glemaitre

Hi @EvgeniDubov. Are there many other split criteria already available? I am only aware of one other which has been discuss but not implemented, Class Confidence Proportion. CCP is similar to Hellinger in that it is designed for imbalanced data. Clearly both can be used for problems which are not imbalanced but I am unaware they have shown utility over Gini or Entropy.

imbalanced-learn covers all common imbalanced classification solutions (undersampling, oversampling, algorithmic, SMOTE) except split criterion methods. It makes sense to add it for completeness. However, if there are enough other split criteria to justify a package that makes sense too.

My major interest is when sklearn will put the #10325 update into their standard release.

#10325 is merged in master and we are heading for a release pretty soon (in the next month).

Looks like imbalanced-learn is mainly focused on data sampling.

We are actually looking at diversify the algorithm until that we are still solving imbalanced issue. Lately I had less time since I focused on scikit-learn and side project but we will make a release after the scikit-learn one and new methods and tools we are happy to include tools there.

@DrEhrfurchtgebietend @glemaitre thanks a lot for your inputs guys.
I've submitted a pull request to scikit-learn-contrib/imbalanced-learn

Hello guys,
thats exactly what im looking for but it fails for me to import.

I copied you're whole repository to ~\site-packages\sklearn\tree
and then tried to compile the setup.py with

python setup.py build_ext --inplace

there was an error that command 'build_ext' does not exist so I tried

python setup.py build_clib

which actually worked but back in my jupyter notebook the command

from hellinger_distance_criterion import HellingerDistanceCriterion

didnt work, so I'm stuck now at this point.

Hope this is the right place to post this issue, or should I rather pass it to stackoverflow?!
I'm already trying like 5 hours but couldn't achieve anything so I hope somebody can resolve this issue.

You might be better off waiting a month or two if you are not in a rush. sklearn 0.20.0 should come out very shortly. This will enable the use of external split criteria. The Hellinger criteria is being developed in imbalanced-learn, have a look here: scikit-learn-contrib/imbalanced-learn#437 Releases of imbalanced-learn normally come out shortly after the new sklearn version they are built on.

Actually i'd like to use this for my thesis but it seems not realizable for me so it seems i have to wait for it and maybe use it in later projects.

@woizaterminator I've a repo with Hellinger criteria implementation and I'm working now on adding it to imbalanced-learn scikit-learn-contrib/imbalanced-learn#437.
So in case you are in a rush and need it in your current project you can find it here EvgeniDubov/hellinger-distance-criterion

Following the comment of @EvgeniDubov, I want it to be in the next release of imbalanced-learn which will follow the release of scikit-learn.
We just need to figure out the generalization of the Hellinger Distance criterion for multi-class problem. Any input would be appreciated ;)

@EvgeniDubov thanks for the implementation! It failed to compile for me in a windows enviroment, because visual studio build tools didnt work for me...compiling it in a linux enviroment worked out for me.

scikit-learn-contrib/imbalanced-learn#437

@EvgeniDubov , I have followed your instruction in https://github.com/EvgeniDubov/hellinger-distance-criterion and successfully installed hellinger distance criterion. However, when I try to run the example in my pycharm or python IDLE, it shows "no module named hellinger_distance_criterion", then I tried the code in "Example" in the python of my terminal console, it works. I don't know if I have missed something, looking forward to your help.

Following the comment of @EvgeniDubov, I want it to be in the next release of imbalanced-learn which will follow the release of scikit-learn.
We just need to figure out the generalization of the Hellinger Distance criterion for multi-class problem. Any input would be appreciated ;)

The implementation by @EvgeniDubov seems to be working consistently in the case of binary classification but in multi-class classification, I am always getting an error. Any help in this regard will be very much appreciated.

when it will be added?

See discussion in #16478 (comment).