If this implements a transform method, it should inherit from TransformerMixin which will give it additional tests.

I'm trying this pull request out just to see if it does reduce memory and am not having any luck. This is the code I'm using, you'll see that my data is pretty conservative:

X = manifold.TSNE(2, method='barnes_hut').fit_transform(np.random.random(size=(40000, 2)))

This eats up 32GB ram in short order, I'm pretty sure I applied the pull request correctly otherwise method='barnes_hut' would throw an error no?

I was under the impression that the barnes hut implementation was much more memory friendly, am I correct?

Edit: this is on Windows 64bit

@gatapia in principle yes, this PR unfortunately not. There is still work to be done to use sparse matrices to reduce the memory footprint. In the current form, it "only" speeds up the computation about one or two orders of magnitude. Any help with going to sparse matrices is very welcome.

I'd really like to have this in 0.17. I removed the transform method because I'm not sure what the API should be there. While the current version could be improved, it gives 10x-100x speedups. I'll post some benchmarks soon.

On 5 classes of the digits its a factor of 2, on all classes it's a factor of 3, for the s-curve it's a factor of 10.

Not sure I'm entirely convinced the new example says a lot, though.

checks pass :) I'm thinking about polishing or removing the faces example, otherwise this should be good.

I'm thinking about polishing or removing the faces example, otherwise this should be good.

+1: either simplify it to only keep the best / fastest performing model or just remove it (a polished version can be re-added later).

In particular it's important to have some explanation that analyze and explain the obtained results: are they good, is the identity or the pose of faces preserved by the 2D embedding? Here it's not very clear to me.

what do you think about the rest?

I think it's already a good improvement upon what we currently have in master. We can work on the memory usage fix in another PR.

+1 for merge once the face example is removed (or simplified and explained).

OK I'll remove it for now and merge.