Seeing lots of failures in py-2.7.

Command python /workspace/appengine/standard/download-appengine-sdk.py /tmp/tmp1midmko3 failed with exit code 1

Build time

lint: 7 min, 54 sec
py-2.7: 19 min, 34 sec
py-3.6: 1 hr, 9 min, 44 sec
py-3.7: 1 hr, 43 min, 28 sec
py-3.8: 1 hr, 11 min, 38 sec

I think we can just go with the default.

I'll fix some flakes then try the full build again.

Now it temporarily includes #4048

All the jobs are just using the default value.

Build time without setting NUM_TEST_WORKERS (defaut values):

lint: 10 min, 36 sec
py-2.7: 21 min, 57 sec
py-3.6: 1 hr, 35 min, 33 sec
py-3.7: 2 hr, 7 min, 50 sec
py-3.8: 1 hr, 17 min, 8 sec

The failure in py-3.8 build is reported at #4054

The builds were rather slow, it might be because of some retries, but it might be because of low number of workers.

The last periodic builds are (btlr is using fixed 30 workers):

lint: 9 min, 38 sec
py-2.7: 22 min, 12 sec
py-3.6: 59 min, 5 sec
py-3.7: 1 hr, 1 min, 56 sec
py-3.8: 1 hr, 0 min, 39 sec

I'll try 10 workers next.

Build time with 10 workers:

lint: 9 min, 24 sec
py-2.7: 20 min, 42 sec
py-3.6: 1 hr, 28 min, 24 sec
py-3.7: 1 hr, 26 min, 5 sec
py-3.8: 1 hr, 18 min, 41 sec

There were no weird error during pip install.
Maybe we can settle to 10 for now.

Seeing a new failure in py-2.7 build:

E     File "/workspace/appengine/standard/migration/incoming/.nox/py-2-7/lib/python2.7/site-packages/rsa/key.py", line 1
E   SyntaxError: Non-ASCII character '\xc3' in file /workspace/appengine/standard/migration/incoming/.nox/py-2-7/lib/python2.7/site-packages/rsa/key.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

I'll update the number of workers for the py-2.7 build to 1. The build is already fast enough without running in parallel.

It seems like the failures in py-2.7 build are deterministic.

https://pypi.org/project/rsa/4.2/#description

Version 4.0 was the last version to support Python 2 and 3.4. Version 4.1 is compatible with Python 3.5+ only.

I think we need to pin rsa-4.0 for some projects.

Update: The new google-auth handles this.

Some tests still fail in py-2.7 build.

Build time with 10 workers:

lint: 10 min, 0 sec
py-2.7: 21 min, 37 sec
py-3.6: 1 hr, 18 min, 40 sec
py-3.7: 1 hr, 23 min, 32 sec
py-3.8: 1 hr, 20 min, 37 sec

There was only one failure in healthcare, which should be fixed by #4064

Restarted py-3.7 build