I don't think it's worthwhile. We'll rely on the tests to detect bugs.

And as you can see, Circle CI is extremely time-consuming these days.

Well, my understanding from our informal poll there was that @amueller , @ogrisel , @rth and I were in favor or running them, hence this PR.

And as you can see, Circle CI is extremely time-consuming these days.

Well, it's 7 minutes here (+3min to upload artifacts), unless I'm missing something. Appveyor takes twice as much. I agree that we rely on tests to actually detect issues, but although I value fast CI, I still wonder if just running all of the examples wouldn't be a simpler solution.

Well, it's 7 minutes here (+3min to upload artifacts), unless I'm missing something. Appveyor takes twice as much. I agree that we rely on tests to actually detect issues, but although I value fast CI, I still wonder if just running all of the examples wouldn't be a simpler solution.

We don't change any code here? Circle CI takes around 40 min in master (and we have 2 jobs).

Well, my understanding from our informal poll there was that @amueller , @ogrisel , @rth and I were in favor or running them, hence this PR.

I somehow recall that there's consensus to rely on the tests :) Let's vote here. I'll vote -1 (maybe +0 if there're too many +1s)

I somehow recall that there's consensus to rely on the tests :) Let's vote here. I'll vote -1 (maybe +0 if there're too many +1s)

It's true that we rely on the tests to detect bugs, but we don't have any systematic way of detecting warnings and issues in the examples to let the authors of PRs fix the examples as well. To me if a PR causes an error in an example, that's an issue needing a fix in that same PR. If you have a better solution to achieve that, I'm happy to hear it.

P.S: #12797 (comment)

Aww, right. I agree 40 min is a lot for each commit in a PR. The summary of options was in #12797 (comment)

Honestly I have not seen to many cases where examples would break on master and not in PR, and that would usually mean that the test coverage is not very good.
I'll change my vote and give a +1 to keep things as they are (i.e. run only changed examples on PR) and all examples on master. I don't think that strictly failing on warning is worth the effort we spend on it. Working to reduce warnings before a release make sense, but it's a never ending effort as soon as one of our dependencies gets a new release.

To me if a PR causes an error in an example, that's an issue needing a fix in that same PR.

Also wouldn't that mean that the PR breaks backward compatibility? I.e. code that worked before now produces an error. This should be very exceptional.

@rth, well, it all started in my brain while I was working on #12654. There are 38 changed files and 112 changed lines in the examples in that PR alone, to fix some warnings. That doesn't really give me a feeling that those cases are rare.

There are 38 changed files and 112 changed lines in the examples in that PR alone, to fix some warnings

There was a significant effort to reduce warnings (and fail on warnings) in tests this summer, but if I remember correctly we didn't quite get to fixing it in examples. If I look at the changes in #12654

• almost half of it is due to changes in dependencies (matplotlib, scikit-image). If we run all examples in CI, and fail on warnings it means that CI in PRs could fail after e.g. a new matplotlib version is released which is not really desirable (unless we pin versions).
• the rest is due to ~3-4 PRs that deprecated some parameters over the last 6+ month. Catching those few warnings doesn't justify making contributors wait 40 min in every single PR for a green CI.

Overall 112 changed lines needed to fix warnings in examples every 6 month or so doesn't sound too bad.

Ok. I'm [kinda] convinced.

Just for the record, this was also discussed in issue #10943.