Hey Gilles. To bad we didn't talk about this. I have a working version of this for classification. But it is not so hard to do. I'll have a look at your implementation later and compare to mine. btw, your pull request can't be merged :-/

@amueller It can now :) But anyway this is nowhere ready. I still have to change the ensemble estimators.

Thanks Gilles - I'm looking forward to study it in more detail; weather this weekend should be really fine in the alps so I might not make it in the next couple of days.

@glouppe this looks pretty cool! can I ask you what your motivation was? I used this for multilabel classification. Your implementation might actually be able to cope with structured class labels.

What I did used a list for y. It did not seem to impact the perfomance much in the single label case and was very flexible. Your method is probably more efficient and more amendable for optimization.

Having completely arbitrary objects as y would be pretty sweet, though (and would be no theoretical problem as long as one can define what node purity is).

wow.. straight from my alma mater (tu-graz) - neat - thanks for the ref.

@pprett do you know hogh forests? they are pretty related and also from somewhere around there I think ;)

ps: sorry for OT

@amueller We are planning to apply that to classify windows of pixels in images.
http://www.montefiore.ulg.ac.be/services/stochastic/pubs/2009/DMWG09/

Regarding my implementation, I consider y as a 2d array (or convert it to that format). A few things may still need to be done though, to convert it appropriately. I'll check that when writing the unit tests.

@glouppe that sounds a look like the paper I cited above :) (but is earlier I think).

@amueller Haha yeah! I am going to carefully read yours I guess ;)

I haven't really read your code but does it do any "extra work" when y is 1d? I think multi-label would be a pretty standart application for this algorithm.

@amueller Well basically it shouldn't. All loops degenerate into single iterations but some (little) overhead is likely.

Oh... I remember why I used lists... in a multi-label setup, each instance can have different numbers of labels. That might not play well with your approach of a fixed length y. We could use 1 of n encodings but that would make it unnecessary slow to work with many classes, I guess. we could also fill the remaining entries with -1 but that also does not really excite my.
If we could address both, the 2d patch and multi-label setting, that would be awesome!

@amueller I handle that :)

@glouppe SWEET! ok I'll keep quite until I read your code ;)

Please not that multi-output is different from multi-labels. I don't know if we are exactly talking about the same thing? In a multi-output classification setting, each output column has its own set of classes and only one of them can be picked at prediction time. In a multi-label setting, as I understand it, you are allowed to predict several classes from the same unique set of classes, which is different. However, you can indeed transform a multi-label problem into a multi-ouput problem using binary encoding (i.e. use n binary outputs, one for each class).

What I wanted to say was that it would be good if we could handle both, multi-output and multi-label. I was afraid the 1-of-n coding might be inefficient if there are many classes but only a few are active at any time.

Well that it cannot handle. :/

However it can handle multi-output classification problems with sets of classes of different sizes at each output. That is what I wanted to say earlier.

Well, maybe doing the binary coding isn't so bad. And doing patch based learning in sklearn is mega-awesome (which I might have not said enough before ;).

I just wanted to say thanks guys especially @glouppe! This has been on my list since I started and I've just not been getting around to it, so I support this 100%!

Very nice example BTW. Except for the few comments above, +1 for merging.

I'm also +1 for merge but if you want I can do another review of the cython code in the evening.

great example and PR!

PS: we should talk to @vene regarding a vbench script which checks for performance regressions in the tree module for future feature requests.

@ogrisel All your comments have been addressed.

@pprett Yes, I am not against a review of the Cython code. Actually, I found a serious bug this morning regarding regressors (segfault and crash). Could you re-run your benchmark on Boston? I guess it'll take longer this time :( (But at least results will be correct)

sure - I'll check

2012/7/4 Gilles Louppe
reply@reply.github.com:

@ogrisel All your comments have been addressed.

@pprett Yes, I am not against a review of the Cython code. Actually, I found a serious bug this morning regarding regressors (segfault and crash). Could you re-run your benchmark on Boston? I guess it'll take longer this time :( (But at least results will be correct)

---

Reply to this email directly or view it on GitHub:
#923 (comment)

Peter Prettenhofer

Any more review? :)

Looks good for me but as I am not a tree expert / user I would rather have @amueller , @bdholt1 or @pprett (or someone else interested in multiple output trees) give it another round of review.

Sorry for the delay - I've been away on holiday. I'd like to give it a
final round if thats possible - perhaps post reviews later this evening?

On 8 July 2012 11:20, Olivier Grisel <
reply@reply.github.com

wrote:

Looks good for me but as I am not a tree expert / user I would rather have
@amueller or @pprett (or someone else interested in multiple output trees)
give it another round of review.

---

Reply to this email directly or view it on GitHub:
#923 (comment)

@bdholt1 Sure!

Thanks for this additional example Brian!

@pprett Waiting for your approval to hit the green button :)

@glouppe Thanks very much for undertaking to implement this, its a very welcome addition! This functionality achieves what mvpart does for R, taking is one step closer to being a complete suite.

@glouppe I re-run the benchmarks on boston - looks very good.

Fit
+--------+----------+-------+-------+
|        |Master    |  MO   |  MOv2 |
+--------+----------+-------+-------+
|Tree(20)|   41.1   |  43.7 | 44.6  |
+--------+----------+-------+-------+
|Tree(1) |     0.6  |  0.712|    .72|
+--------+----------+-------+-------+
|RF      |      338 |  296  |  321  |
+--------+----------+-------+-------+
|GBRT    |      90  |  109  |  112  | 
+--------+----------+-------+-------+

Predict
+--------+-------------+-------+------+
|        |Master       |MO     |MO    |
+--------+-------------+-------+------+
|Tree(20)|      0.09   |  0.1  | 0.1  |
+--------+-------------+-------+------+
|Tree(1) |     0.031   |  0.034|0.037 |
+--------+-------------+-------+------+
|RF      |      30     |  1.1  | 1.2  |
+--------+-------------+-------+------+
|GBRT    |      0.775  |0.768  | 0.82 |
+--------+-------------+-------+------+

@glouppe there are two formatting errors in the doctests (tree.rst); apart from that I'm +1

great work - thx!

Thank you all for the reviews! I merge :)

Great Work. Thanks a lot!