Notes from:

• webserver may not be allowed to listen on default port
• may be able to attach failed test images to junit xml
• failing that, there are uploads steps to see failed images

no guesses yet on the failing gui test...

@tacaswell

2018-11-02T20:55:59.7604603Z [gw0] linux -- Python 3.5.5 /opt/hostedtoolcache/Python/3.5.5/x64/python
2018-11-02T20:55:59.7604713Z 
2018-11-02T20:55:59.7605152Z     @pytest.mark.skipif(os.name == "nt", reason="Cannot send SIGINT on Windows.")
2018-11-02T20:55:59.7606710Z     def test_webagg():
2018-11-02T20:55:59.7610820Z         pytest.mark.skipif('SYSTEM_TEAMFOUNDATIONCOLLECTIONURI' in os.environ)
2018-11-02T20:55:59.7611019Z         pytest.importorskip("tornado")
2018-11-02T20:55:59.7612108Z         proc = subprocess.Popen([sys.executable, "-c", _test_script],
2018-11-02T20:55:59.7613937Z                                 env={**os.environ, "MPLBACKEND": "webagg"})
2018-11-02T20:55:59.7615213Z         url = "http://{}:{}".format(
2018-11-02T20:55:59.7616952Z             mpl.rcParams["webagg.address"], mpl.rcParams["webagg.port"])
2018-11-02T20:55:59.7618457Z         timeout = time.perf_counter() + _test_timeout
2018-11-02T20:55:59.7619405Z         while True:
2018-11-02T20:55:59.7620863Z             try:
2018-11-02T20:55:59.7622102Z                 retcode = proc.poll()
2018-11-02T20:55:59.7623560Z                 # check that the subprocess for the server is not dead
2018-11-02T20:55:59.7624752Z >               assert retcode is None
2018-11-02T20:55:59.7627170Z E               assert 0 is None

Would be appropriate to say assert not retcode here?

The issue is that for some reason the webserver starts and then immediately exits. If the process is still running poll return None, if it is exited, it return the returncode (which in this case is 0).

I can not see to identify the we are on azure to skip these two....

I've been digging around trying to see if we can pass an e.g., --is-azure flag to the command line and check for that as a global variable/fixture

something like this maybe. need to tinker a bit
pytest-dev/pytest#2358

I found a more fundamental problem with my approach... (see the last commit).

If this passes I am going to self squash-merge.

It passes!

@meeseeksdev backport to v3.0.x

@meeseeksdev backport to v2.2.x

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

$ git checkout v2.2.x
$ git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

$ git cherry-pick -m1 531bd02409ba5efd75ec13f713f6faf6974199d4

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

$ git commit -am 'Backport PR #12617: Set up CI with Azure Pipelines'

4. Push to a named branch :

git push YOURFORK v2.2.x:auto-backport-of-pr-12617-on-v2.2.x

5. Create a PR against branch v2.2.x, I would have named this PR:

"Backport PR #12617 on branch v2.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to suggest an improvement.

Backporting this so that CI passes on the old branchs (I don't see how to restrict azure to just the master branch).

So, does this mean we don't need the Travis checks anymore?

It doesn't seem like it actually ran on the backport, so maybe you don't need it?

As we can turn this on by-branch lets not drag this back to 2.2.x unless someone feel very motivated.