[MRG+2] ENH : SAGA support for LogisticRegression (+ L1 support), Ridge #8446

arthurmensch · 2017-02-23T18:02:07Z

This PR proposes slight adaptations of the existing sag_fast.pyx module to be able to use SAGA in addition to the SAG algorithm. This is the way to go if we want to propose enet/l1 penalty for fast incremental solvers in for ridge and logistic regression.

A SAGA implementation is already available in lightning, which is adapted from the original paper. This one is slightly different as it is built around understanding SAGA update as a "corrected" version of SAG.
I believe it would also be possible to slightly adapt the module to have SVRG in addition to SAGA, if this interests people.

I tried to keep the changes made to sag_fast.pyx as scarse as possible. I reckon that the sag_fast.pyx module could be made a little more readable using 2d memoryviews instead of using strided pointers everywhere. For further work.

For the moment I adapted the test_logistic.py file to ensure correctness of the algorithm, but the saga algorithm should be tested within test_sag.py.

SAGA paper

TODO

GaelVaroquaux · 2017-02-23T18:04:55Z

In the todo, you should probably add a benchmark that would show the benefit.

GaelVaroquaux · 2017-02-23T18:04:56Z

By the way: I am quite excited about this!

arthurmensch · 2017-02-23T18:06:29Z

Indeed quite a few benchmarks would be necessary. Added !

arthurmensch · 2017-02-23T18:09:12Z

sklearn/linear_model/logistic.py

-from ..externals import six
-from ..metrics import SCORERS
+from ..utils.optimize import newton_cg
+from ..utils.validation import check_X_y


I guessed my editor just put these automatically in alphabetic order, I can revert if necessary

GaelVaroquaux · 2017-02-23T18:10:46Z

I guessed my editor just put these automatically in alphabetic order, I can revert if necessary

It would be nice. If files start being reformated by editors, we will have a lot of conflicts.

codecov · 2017-02-23T20:24:22Z

Codecov Report

Merging #8446 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #8446      +/-   ##
==========================================
+ Coverage   95.47%   95.48%   +<.01%     
==========================================
  Files         342      342              
  Lines       60907    61007     +100     
==========================================
+ Hits        58154    58255     +101     
+ Misses       2753     2752       -1

Impacted Files	Coverage Δ
sklearn/linear_model/tests/test_sag.py	`98.6% <100%> (+0.07%)`	✅
sklearn/linear_model/sag.py	`94.36% <100%> (+0.52%)`	✅
sklearn/linear_model/tests/test_logistic.py	`100% <100%> (ø)`	✅
sklearn/linear_model/logistic.py	`97.65% <100%> (+0.03%)`	✅
sklearn/linear_model/tests/test_ridge.py	`100% <100%> (ø)`	✅
sklearn/linear_model/ridge.py	`93.88% <100%> (ø)`	✅
sklearn/linear_model/coordinate_descent.py	`96.94% <0%> (ø)`	✅
sklearn/tree/tree.py	`98.41% <0%> (ø)`	✅
sklearn/metrics/classification.py	`97.77% <0%> (ø)`	✅
sklearn/decomposition/tests/test_pca.py	`100% <0%> (ø)`	✅
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcb1403...2a5c4bf. Read the comment docs.

arthurmensch · 2017-02-23T22:17:53Z

A quick benchmark :

Classification performance:
===========================
Classifier               train-time   test-time   error-rate
------------------------------------------------------------
LinearRegression-SAGA        17.25s       0.07s       0.0810
LinearRegression-SAG         20.64s       0.10s       0.0824

arthurmensch · 2017-02-24T14:47:17Z

I added some code to do benchmarks, which should be removed for merging.

With the conservative auto step size that is used, SAGA performs better in the first epochs and SAG gets better for finer convergence (a behavior already observed in the litterature). With more aggressive stepsizes, SAGA performs better than SAG. We could use line-search as it tends to produce better results, it is used in sgd in scikit-learn ?

GaelVaroquaux · 2017-02-24T14:51:44Z

We could use linesearch as it tends to produce better results, it is used somewhere in scikit-learn ?

I don't think so. It's used in the LBFGS code, but that's not scikit-learn.

agramfort · 2017-02-26T16:10:25Z

@arthurmensch can you show a benchmark with $log10(f(x^k) - f(x^*))$ on y axis and time on x axis?
I suspect marginal improvements for L2 log reg so it would need a convincing figure on L1 reg logistic so a comparison with liblinear that we use presently.

arthurmensch · 2017-02-27T14:54:24Z

@agramfort can you recall a dataset on which SAG would beat liblinear with l2 penalty ?

agramfort · 2017-02-27T15:30:18Z

rcv1 or covertype?

arthurmensch · 2017-02-28T22:03:58Z

I did some benchmarks against lightning for the moment, on rcv1. For some reason we are 10x faster than lightning with L1 penalty, any thought @fabianp ?

l2 logistic:

l1 logistic:

Credits for the shortcut in the composition of prox operators goes to @fabianp and @mblondel, but I think it looks a bit cleaner that way.

I will post benchmarks against liblinear ASAP.

GaelVaroquaux · 2017-02-28T22:08:59Z

This is looking great. And the fun part is that it seems that reimplementing code and comparing teaches us a lot, even when the two implementations are done by people that share so much.

arthurmensch · 2017-02-28T22:40:13Z

For single class l1/l2 logistic regression liblinear is hard to beat. But for multiclass, speed improvements look great:

There is a discrepency in the final accuracy as both model are not the same (the loss is different, and liblinear penalizes the intercept, but I reckon they perform the same with correct cross validation.

Two-class problem (there is a problem there as the final training score should not be that different).

One of the advantage of using SAGA instead of liblinear is that we can perform CV with memory-mapped data, which can be crucial when datasets are huge.

mblondel · 2017-02-28T23:30:09Z

It could be interesting to see if both implementations return more or less the same weight vectors.

I remember it took @fabianp and @zermelozf several iterations to implement the lazy updates correctly. We have a naive Pure python implementation in our tests to ensure correctness:

https://github.com/scikit-learn-contrib/lightning/blob/master/lightning/impl/tests/test_sag.py#L85

arthurmensch · 2017-03-01T10:55:07Z

These are the right benches liblinear vs lightning vs sklearn saga, for 1/20 of rcv1.

SAGA sklearn is better for multitarget regression for l2 and l1
SAGA sklearn is better for single target regression for l2, and a little slower than liblinear with l1
SAGA sklearn and lightning are roughtly on par with l2 regularisation, single-target
SAGA sklearn is ten times faster than lightning with l1 regularisation, single-target. This is surprising

I am currently running the benches on the whole dataset, to see if saga does not get better than liblinear at some point.

The benchmark file that is in this PR does not use callbaks or any hacks, so I think it can stay in the repo for future reference.

GaelVaroquaux · 2017-03-01T13:59:47Z

I think that liblinear beat saga in the l1 case because coordinate descent methods are a very efficient when the coefficients are sparse if there is a good heuristic to select the coeffficient.

arthurmensch · 2017-03-02T10:07:14Z

Should I try to add a solver for Lasso ? For the moment scikit-learn has quite a variety of Lasso solvers (Lasso, LassoLars, CV versions, IC versions, RandomizedLasso) so I am not sure where to put it. In Lasso, adding a solver option ?

TomDLT

Nice work ! The code looks good so far.
I still need to understand the JIT prox update.

Use minibatches ? I cannot recall whether this is actually interesting.

Mark Schmidt says it's interesting in SAG (slide 73), yet I am not sure we need it in scikit-learn.

TomDLT · 2017-03-02T10:38:53Z

sklearn/linear_model/logistic.py

-
+            if penalty == 'l1':
+                if solver == 'sag':
+                    raise ValueError("Unsupported penalty. Use `saga` instead.")


This is never reached, thanks to _check_solver_option, isn't it?

TomDLT · 2017-03-02T10:42:15Z

sklearn/linear_model/sag.py



-def get_auto_step_size(max_squared_sum, alpha_scaled, loss, fit_intercept):
+def get_auto_step_size(max_squared_sum, alpha_scaled, loss, fit_intercept,
+                       n_samples=None,


update docstring

TomDLT · 2017-03-02T10:56:50Z

sklearn/linear_model/sag_fast.pyx

+
+
+
+cdef double lagged_update(double* weights, double wscale, int xnnz,


update the docstring description

TomDLT · 2017-03-02T12:58:29Z

sklearn/linear_model/sag_fast.pyx

+        else:
+            for class_ind in range(n_classes):
+                idx = f_idx + class_ind
+                if fabs(sum_gradient[idx] * cum_sum) < cum_sum_prox:


can you explain this? and maybe add some comments

This is a nice trick from lightning: instead of enrolling the whole delayed softmax(softmax(softmax(w - grad_update) - grad_update ...) in the loop below, we factorize it as we do not cross the non-linearity due to the softmax. There is no academical reference for this, but we should indeed add some comments.

It would be nice if there was a small blog post with the derivations for this on the web. @fabianp sent me an unfinished draft for this but I gathered it was to stay a draft.

If you do not have time I can write a small blog post wit mathematical derivation for this part, with due reference to @fabianp @zermelozf @mblondel. Then we can reference it in comment.

That would be awesome. I'm sending you the .tex sources of that draft so you can use it as you please.

Even without the full blog post with the derivation it would be great to have an inline comment to state:

what you said in https://github.com/scikit-learn/scikit-learn/pull/8446/files#r103928318

give credits to the lightning implementation

TomDLT · 2017-03-02T13:01:19Z

sklearn/linear_model/sag_fast.pyx

+    if reset:
+        cumulative_sums[sample_itr - 1] = 0.0
+        if prox:
+            cumulative_sums_prox[sample_itr - 1] = 0.0

    # reset wscale to 1.0


You should return void and return 1.0 only in scale_weights

TomDLT · 2017-03-02T13:02:51Z

sklearn/linear_model/sag_fast.pyx

+    cdef np.ndarray[double, ndim=1] cumulative_sums_prox_array
+    cdef double* cumulative_sums_prox
+
+    cdef bint prox = beta > 0


it could be safe to check that saga is True

TomDLT · 2017-03-02T13:07:22Z

sklearn/linear_model/sag.py

    else:
        raise ValueError("Unknown loss function for SAG solver, got %s "
                         "instead of 'log' or 'squared'" % loss)
+    if is_saga:
+        mun = min(2 * n_samples * alpha_scaled, L)


where does it come from?

SAGA original paper: proofs of convergence requires step_size < 1 / 3L or step_size < 1 / (2(L + mu n), where mu is the strong convexity modulus of the objective. We could use 1 / 3L but this is more optimal in the low dimensional regime. I will add a reference. By the way SAG use 1 / L whereas the only step size for which proofs are available is 1 / 16 L. I think this is a sound heuristic but we should add a reference as well. 1 / L is also a good heuristic for SAGA in most cases, but it actually make one test fail in the test suite :P We could allow the user to specify the step size, but it would complexify the API.

Ok
For SAG, I used the recommendation of Mark Schmidt: (slide 65)

TomDLT · 2017-03-02T13:08:20Z

sklearn/linear_model/sag.py

@@ -261,26 +272,42 @@ def sag_solver(X, y, sample_weight=None, loss='log', alpha=1.,
    if max_squared_sum is None:
        max_squared_sum = row_norms(X, squared=True).max()
    step_size = get_auto_step_size(max_squared_sum, alpha_scaled, loss,


should the step size depend on beta_scaled in the L1 case?

No, it depends on L which is the Lipschitz constant of the gradient of the smooth objective part (i.e logistic + optional l2).

oh yes of course

arthurmensch · 2017-03-02T16:09:28Z

Should we add an example for SAGA + some narrative documentation ? I suggest multinomial logistic + l1 on rcv1, with comparison against one versus all with liblinear, although this would take a few minutes on the whole dataset (because of ovr).

I suggest we work on merging this PR before working on using SAGA for Lasso.

There is also a question regarding whether we should propose both sag and saga to the end-user, or deprecate sag ? @GaelVaroquaux.

arthurmensch · 2017-03-02T16:58:40Z

ping @TomDLT @agramfort @GaelVaroquaux I think this is ready for review. Reviews welcome from @fabianp and @mblondel for the core code if you have time.

A few UX questions to answer before I move on :

Do we agree on a multinomial + L1 example on rcv1, with comparison with ovr + liblinear ?
Do we deprecate sag solver or keep both sag and saga ? Is sag solver useful in any way compared to saga @fabianp ?
Do we add an l1_ratio to LogisticRegression as we now can ? C + l1_ratio sounds a bit original but it is still meaningful.
Naming is now a bit fishy (sag_solver for saga). I guess we can keep it that way for the moment, but it will require refactoring if we add for instance an svrg solver to the code base.

I reckon adding l1_ratio can be made in a new PR to keep this one minimalistic.

arthurmensch · 2017-03-03T12:39:15Z

On the whole rcv1 dataset, single target with high regularization, saga is faster than liblinear: C=0.1

This is not true for low regularization : C=1

This is expected (with high regularization the soft thresholding shortcut trick works more often).

TomDLT · 2017-03-03T12:45:41Z

How does SAGA perform on small datasets (e.g. iris)?
It would be great if it would compete with liblinear, so we can change the default solver in LogisticRegression.
The regularization of the intercept in liblinear is often confusing for users.

GaelVaroquaux · 2017-03-03T13:14:50Z

We will need a very clear paragraph in the documentation that explains which solver to choose when.

ogrisel · 2017-03-20T10:55:44Z

examples/linear_model/plot_sparse_logistic_regression_20newsgroups.py

@@ -0,0 +1,114 @@
+"""
+=====================================================
+Multiclass logisitic regression on newgroups20


Please also put "sparse" in the title.

ogrisel · 2017-03-20T10:59:02Z

examples/linear_model/plot_sparse_logistic_regression_20newsgroups.py

+features to zero. This is good if the goal is to extract the strongly
+discriminative vocabulary of each class. If the goal is to get the best
+predictive accuracy, it is better to use the non sparsity-inducing
+l2 penalty instead.


I think you should also mention univariate feature selection as an alternative way to extract sparse discriminative vocabularies.

Maybe you could even extend the example by adding a pipeline of a sparse uni variate feature selection model + l2 penalized logistic regression to showcase a classification model with similar sparsity level as the l1 penalized variant.

ogrisel · 2017-03-20T11:01:17Z

examples/linear_model/plot_sparse_logistic_regression_mnist.py

+Performance of multinomial logistic regression with
+L1 penalty. We use the SAGA algorithm for this purpose, which is fast. Test
+accuracy reaches > 0.8, while weight vectors remains *sparse* and
+*interpretable*.


Please add a note such that:

Note that this accuracy is far below what can be reached by an non-penalized linear model (I think ~0.93) but this should be checked and even more far below the accuracy non linear models such as a multi layer perceptron (0.98+).

ogrisel · 2017-03-21T12:53:31Z

sklearn/linear_model/sag_fast.pyx

        np.ndarray[double, ndim=2, mode='c'] sum_gradient_init,
        np.ndarray[double, ndim=2, mode='c'] gradient_memory_init,
        np.ndarray[bint, ndim=1, mode='c'] seen_init,
        int num_seen,
        bint fit_intercept,
        np.ndarray[double, ndim=1, mode='c'] intercept_sum_gradient_init,
        double intercept_decay,
+        bint saga,
        bint verbose):
    """Stochastic Average Gradient (SAG) solver.


Please update this docstring to make it explicit that this function implements both SAG and SAGA.

ogrisel · 2017-03-22T16:32:09Z

I pushed better checks for the l1 penalty logistic regression tests. I still want to review other parts of the code / tests but maybe we should do the Lasso part in a separate PR.

ogrisel

I did a review and pushed some fixes (mostly missing updates in the documentation). I think we can merge this PR without waiting for elasticnet penalty and integration in the Lasso* classes.

TomDLT

LGTM except for minor nitpicks

I think we can merge this PR without waiting for elasticnet penalty and integration in the Lasso* classes.

I agree

TomDLT · 2017-03-27T16:47:32Z

doc/modules/linear_model.rst

+Very Large dataset (`n_samples`)   "sag" or "saga"
+=================================  =====================================
+
+The "saga" solver is almost always a the best choice. The "liblinear"


TomDLT · 2017-03-27T16:49:26Z

examples/linear_model/plot_sparse_logistic_regression_20newsgroups.py

+regression yields more accurate results and is faster to train on the larger
+scale dataset.
+
+Here we use the l1 sparsity that trims the weights of no to informative


-> of not informative

TomDLT · 2017-03-27T16:50:43Z

sklearn/linear_model/logistic.py

@@ -967,6 +976,9 @@ class LogisticRegression(BaseEstimator, LinearClassifierMixin,
        Used to specify the norm used in the penalization. The 'newton-cg',
        'sag' and 'lbfgs' solvers support only l2 penalties.

+        .. versionadded:: 0.19
+           l1 penalty with SAGA solver (allowing 'multinomial + L1)


TomDLT · 2017-03-27T16:52:52Z

sklearn/linear_model/sag_fast.pyx

@@ -6,16 +6,16 @@
 #          Tom Dupre la Tour <tom.dupre-la-tour@m4x.org>
 #


TomDLT · 2017-03-27T16:54:53Z

sklearn/linear_model/tests/test_logistic.py

@@ -860,6 +895,71 @@ def test_logreg_intercept_scaling_zero():
    assert_equal(clf.intercept_, 0.)


+def test_logreg_l1():
+    # Because liblinear penalizes the intercept and saga does not, we do


we do not

TomDLT · 2017-03-27T16:55:21Z

sklearn/linear_model/tests/test_logistic.py

+
+
+def test_logreg_l1_sparse_data():
+    # Because liblinear penalizes the intercept and saga does not, we do


we do not

ogrisel · 2017-03-27T18:41:52Z

I rebased an squashed everything down to a single commit. If CI is still green, let's merge.

ogrisel · 2017-03-27T19:39:48Z

Merged! Thanks @arthurmensch!

arthurmensch · 2017-03-27T19:54:03Z

Great !! Thanks for the reviews and edits !

…

On Mar 27, 2017 9:40 PM, "Olivier Grisel" ***@***.***> wrote: Merged! Thanks @arthurmensch <https://github.com/arthurmensch>! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8446 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD48Yyw-V14iu9Is-e8buk2M0o5Kb5prks5rqBCPgaJpZM4MKTET> .

TomDLT · 2017-03-27T20:43:04Z

🍻

GaelVaroquaux · 2017-03-28T06:03:17Z

Awesome!!!!!

jnothman · 2017-03-28T08:53:43Z

Wow!

…

On 28 Mar 2017 5:03 pm, "Gael Varoquaux" ***@***.***> wrote: Awesome!!!!! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8446 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6wXdVv4--QFr1ufiVQkOVO36oULrks5rqKKngaJpZM4MKTET> .

agramfort · 2017-03-28T10:26:57Z

🍻 !!!

fabianp · 2017-03-28T13:09:40Z

congrats @arthurmensch and co. I think this is a great example of development that started in scikit-learn-contrib and (with a lot of work and improvements) ended upstream.

arthurmensch commented Feb 23, 2017

View reviewed changes

arthurmensch force-pushed the saga branch from 0e756ee to a78b0d0 Compare February 23, 2017 20:26

arthurmensch changed the title ~~[WIP] SAGA support for LogisticRegression and Ridge~~ [MRG] SAGA support for LogisticRegression and Ridge Feb 28, 2017

TomDLT reviewed Mar 2, 2017

View reviewed changes

arthurmensch changed the title ~~[MRG] SAGA support for LogisticRegression and Ridge~~ [MRG] ENH : SAGA support for LogisticRegression, Ridge and Lasso (+ l1 support) Mar 2, 2017

arthurmensch changed the title ~~[MRG] ENH : SAGA support for LogisticRegression, Ridge and Lasso (+ l1 support)~~ [MRG] ENH : SAGA support for LogisticRegression, Ridge and Lasso (+ L1 support) Mar 2, 2017

jnothman mentioned this pull request Mar 3, 2017

linear classifier with elasticnet penalty #8288

Closed

ogrisel reviewed Mar 20, 2017

View reviewed changes

ogrisel reviewed Mar 21, 2017

View reviewed changes

ogrisel approved these changes Mar 27, 2017

View reviewed changes

ogrisel changed the title ~~[MRG] ENH : SAGA support for LogisticRegression, Ridge and Lasso (+ L1 support)~~ [MRG+1] ENH : SAGA support for LogisticRegression, Ridge and Lasso (+ L1 support) Mar 27, 2017

ogrisel changed the title ~~[MRG+1] ENH : SAGA support for LogisticRegression, Ridge and Lasso (+ L1 support)~~ [MRG+1] ENH : SAGA support for LogisticRegression (+ L1 support), Ridge Mar 27, 2017

TomDLT approved these changes Mar 27, 2017

View reviewed changes

Add SAGA solver for LogisticRegression and Ridge

cf4c2e2

ogrisel force-pushed the saga branch from a65f60e to cf4c2e2 Compare March 27, 2017 18:40

ogrisel changed the title ~~[MRG+1] ENH : SAGA support for LogisticRegression (+ L1 support), Ridge~~ [MRG+2] ENH : SAGA support for LogisticRegression (+ L1 support), Ridge Mar 27, 2017

ogrisel merged commit 5147fd0 into scikit-learn:master Mar 27, 2017

TomDLT mentioned this pull request Apr 26, 2017

LogisticRegression does not have a solver for multi_class='multinomial' and penalty='l1' #8797

Closed

massich pushed a commit to massich/scikit-learn that referenced this pull request Apr 26, 2017

Add SAGA solver for LogisticRegression and Ridge (scikit-learn#8446)

34a249b

Sundrique pushed a commit to Sundrique/scikit-learn that referenced this pull request Jun 14, 2017

Add SAGA solver for LogisticRegression and Ridge (scikit-learn#8446)

4af9b07

NelleV pushed a commit to NelleV/scikit-learn that referenced this pull request Aug 11, 2017

Add SAGA solver for LogisticRegression and Ridge (scikit-learn#8446)

65ec7e6

paulha pushed a commit to paulha/scikit-learn that referenced this pull request Aug 19, 2017

Add SAGA solver for LogisticRegression and Ridge (scikit-learn#8446)

6d60807

maskani-moh pushed a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017

Add SAGA solver for LogisticRegression and Ridge (scikit-learn#8446)

da8259a

lesteve mentioned this pull request Nov 16, 2017

beta parameter not documented in sag_solver #10150

Closed

jwjohnson314 pushed a commit to jwjohnson314/scikit-learn that referenced this pull request Dec 18, 2017

Add SAGA solver for LogisticRegression and Ridge (scikit-learn#8446)

9a9060b

lesteve mentioned this pull request Dec 19, 2019

[MRG] Minor fixes in plot_sparse_logistic_regression_20newsgroups.py #15925

Merged




		cdef double lagged_update(double* weights, double wscale, int xnnz,

		@@ -6,16 +6,16 @@
		# Tom Dupre la Tour <tom.dupre-la-tour@m4x.org>
		#



		def test_logreg_l1_sparse_data():
		# Because liblinear penalizes the intercept and saga does not, we do

[MRG+2] ENH : SAGA support for LogisticRegression (+ L1 support), Ridge #8446

[MRG+2] ENH : SAGA support for LogisticRegression (+ L1 support), Ridge #8446

Conversation

arthurmensch commented Feb 23, 2017 • edited Loading

TODO

GaelVaroquaux commented Feb 23, 2017 via email

GaelVaroquaux commented Feb 23, 2017 via email

arthurmensch commented Feb 23, 2017

Choose a reason for hiding this comment

GaelVaroquaux commented Feb 23, 2017 via email

codecov bot commented Feb 23, 2017 • edited Loading

Codecov Report

arthurmensch commented Feb 23, 2017

arthurmensch commented Feb 24, 2017 • edited Loading

GaelVaroquaux commented Feb 24, 2017 via email

agramfort commented Feb 26, 2017

arthurmensch commented Feb 27, 2017

agramfort commented Feb 27, 2017 via email

arthurmensch commented Feb 28, 2017

GaelVaroquaux commented Feb 28, 2017

arthurmensch commented Feb 28, 2017 • edited Loading

mblondel commented Feb 28, 2017

arthurmensch commented Mar 1, 2017 • edited Loading

GaelVaroquaux commented Mar 1, 2017 via email

arthurmensch commented Mar 2, 2017 • edited Loading

TomDLT left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arthurmensch Mar 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arthurmensch Mar 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arthurmensch commented Mar 2, 2017 • edited Loading

arthurmensch commented Mar 2, 2017 • edited Loading

arthurmensch commented Mar 3, 2017 • edited Loading

TomDLT commented Mar 3, 2017

GaelVaroquaux commented Mar 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogrisel Mar 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogrisel commented Mar 22, 2017

ogrisel left a comment

Choose a reason for hiding this comment

TomDLT left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogrisel commented Mar 27, 2017

ogrisel commented Mar 27, 2017

arthurmensch commented Mar 27, 2017 via email

TomDLT commented Mar 27, 2017

GaelVaroquaux commented Mar 28, 2017 via email

jnothman commented Mar 28, 2017 via email

agramfort commented Mar 28, 2017 via email

fabianp commented Mar 28, 2017

arthurmensch commented Feb 23, 2017 •

edited

Loading

codecov bot commented Feb 23, 2017 •

edited

Loading

arthurmensch commented Feb 24, 2017 •

edited

Loading

arthurmensch commented Feb 28, 2017 •

edited

Loading

arthurmensch commented Mar 1, 2017 •

edited

Loading

arthurmensch commented Mar 2, 2017 •

edited

Loading

arthurmensch Mar 2, 2017 •

edited

Loading

arthurmensch Mar 2, 2017 •

edited

Loading

arthurmensch commented Mar 2, 2017 •

edited

Loading

arthurmensch commented Mar 2, 2017 •

edited

Loading

arthurmensch commented Mar 3, 2017 •

edited

Loading

ogrisel Mar 20, 2017 •

edited

Loading

TomDLT left a comment •

edited

Loading