[MRG] Fix min-dependencies doc build #14515

lesteve · 2019-07-30T07:21:18Z

This has been failing on master since yesterday, see this for example.

Problem originally seen in #14507 (comment).

The work-around is to add free conda channel if conda environment creation fails.

rth · 2019-07-30T11:30:54Z

build_tools/circle/build_doc.sh

+# are using them for our min-dependencies doc generation. See
+# https://www.anaconda.com/why-we-removed-the-free-channel-in-conda-4-7/ for
+# more details.
+conda_command || (conda config --set restore_free_channel true && conda_command)


There is a number of reasons why the first conda_command could fail. Also for the doc-minimal job it would run the dependency solver twice.

I think it would be better to just add a -c conda-forge above. It should work at least for the doc-minimal versions.

This was a first attempt at solving the problem and I wanted to make sure this was actually solving it. More than open to better suggestions!

I was thinking using CIRCLE_JOB (https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables) to detect that we are in the min-dependencies doc build would be the simplest solution but I haven't tried yet.

Using conda-forge could be another work-around. One of the problem is that the exact version of the packages we ask for are not in conda-forge.

CIRCLE_JOB to detect that we are in the min-dependencies doc build would be the simplest solution but I haven't tried yet

Yeah, that would work as well.

One of the problem is that the exact version of the packages we ask for are not in conda-forge.

Are you sure? For the minimal or the default build?

The minimal seems to resolve fine,

conda create -n test-env -c conda-forge python="${PYTHON_VERSION:-*}" numpy="${NUMPY_VERSION:-*}" scipy="${SCIPY_VERSION:-*}" "cython>=0.28.5" pytest coverage matplotlib="${MATPLOTLIB_VERSION:-*}" sphinx=2.1.2 pillow scikit-image="${SCIKIT_IMAGE_VERSION:-*}" pandas="${PANDAS_VERSION:-*}" joblib Collecting package metadata: done Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.6.14 latest version: 4.7.10 Please update conda by running $ conda update -n base -c defaults conda ## Package Plan ## environment location: /home/rth/.miniconda3/envs/test-env added / updated specs: - coverage - cython[version='>=0.28.5'] - joblib - matplotlib=1.5.1 - numpy=1.11.0 - pandas - pillow - pytest - python=3.5 - scikit-image=0.12.3 - scipy=0.17.0 - sphinx=2.1.2 The following packages will be downloaded: package | build ---------------------------|----------------- alabaster-0.7.12 | py_0 15 KB conda-forge asn1crypto-0.24.0 | py35_3 155 KB conda-forge atomicwrites-1.3.0 | py_0 9 KB conda-forge attrs-19.1.0 | py_0 32 KB conda-forge babel-2.7.0 | py_0 5.9 MB conda-forge blas-1.1 | openblas 1 KB conda-forge bokeh-0.13.0 | py35_0 5.1 MB conda-forge bzip2-1.0.8 | h516909a_0 396 KB conda-forge ca-certificates-2019.6.16 | hecc5488_0 145 KB conda-forge certifi-2018.8.24 | py35_1001 139 KB conda-forge cffi-1.11.5 | py35h5e8e0c9_1 406 KB conda-forge chardet-3.0.4 | py35_3 189 KB conda-forge click-7.0 | py_0 61 KB conda-forge cloudpickle-1.2.1 | py_0 22 KB conda-forge coverage-4.5.1 | py35h470a237_1 240 KB conda-forge cryptography-2.3.1 | py35hdffb7b8_0 959 KB conda-forge cryptography-vectors-2.3.1 | py35_0 30.5 MB conda-forge cycler-0.10.0 | py_1 8 KB conda-forge cython-0.28.5 | py35hfc679d8_0 8.8 MB conda-forge cytoolz-0.9.0.1 | py35h470a237_0 1.3 MB conda-forge dask-0.19.2 | py_0 3 KB conda-forge dask-core-0.19.2 | py_0 492 KB conda-forge decorator-4.4.0 | py_0 11 KB conda-forge distributed-1.23.2 | py35_1 827 KB conda-forge docutils-0.14 | py35_1 707 KB conda-forge fontconfig-2.11.1 | 6 405 KB freetype-2.5.5 | 2 2.5 MB heapdict-1.0.0 | py35_1000 7 KB conda-forge icu-64.2 | he1b5a44_0 12.6 MB conda-forge idna-2.7 | py35_2 132 KB conda-forge imagesize-1.1.0 | py_0 6 KB conda-forge jinja2-2.10.1 | py_0 91 KB conda-forge joblib-0.13.2 | py_0 180 KB conda-forge jpeg-9c | h14c3975_1001 251 KB conda-forge libffi-3.2.1 | he1b5a44_1006 46 KB conda-forge libgfortran-3.0.0 | 1 281 KB conda-forge libiconv-1.15 | h516909a_1005 2.0 MB conda-forge libpng-1.6.37 | hed695b0_0 343 KB conda-forge libtiff-4.0.6 | 7 507 KB conda-forge libxml2-2.9.9 | hee79883_2 1.3 MB conda-forge locket-0.2.0 | py_2 6 KB conda-forge markupsafe-1.0 | py35h470a237_1 33 KB conda-forge matplotlib-1.5.1 | np111py35_0 8.5 MB more-itertools-7.2.0 | py_0 48 KB conda-forge msgpack-python-0.5.6 | py35h2d50403_3 300 KB conda-forge ncurses-6.1 | hf484d3e_1002 1.3 MB conda-forge networkx-2.3 | py_0 1.1 MB conda-forge numpy-1.11.0 |py35_blas_openblas_100 6.8 MB conda-forge openblas-0.2.14 | 4 3.6 MB openssl-1.0.2r | h14c3975_0 3.1 MB conda-forge packaging-19.0 | py_0 23 KB conda-forge pandas-0.23.4 | py35hf8a1672_0 27.2 MB conda-forge partd-1.0.0 | py_0 16 KB conda-forge pathlib2-2.3.2 | py35_0 31 KB conda-forge pillow-3.2.0 | py35_1 817 KB conda-forge pip-18.0 | py35_1001 1.8 MB conda-forge pluggy-0.11.0 | py_0 17 KB conda-forge psutil-5.4.7 |py35h14c3975_1001 307 KB conda-forge py-1.8.0 | py_0 65 KB conda-forge pycparser-2.19 | py_0 87 KB conda-forge pygments-2.4.2 | py_0 661 KB conda-forge pyopenssl-18.0.0 | py35_0 80 KB conda-forge pyparsing-2.4.1.1 | py_0 57 KB conda-forge pyqt-4.11.4 | py35_3 3.5 MB conda-forge pysocks-1.6.8 | py35_2 22 KB conda-forge pytest-3.8.1 | py35_0 316 KB conda-forge python-3.5.5 | h5001a0f_2 20.2 MB conda-forge python-dateutil-2.8.0 | py_0 219 KB conda-forge pytz-2019.1 | py_0 227 KB conda-forge pyyaml-3.13 | py35h470a237_1 442 KB conda-forge qt-4.8.7 | 3 34.1 MB readline-7.0 | hf8c457e_1001 391 KB conda-forge requests-2.19.1 | py35_1 94 KB conda-forge scikit-image-0.12.3 | np111py35_1 27.4 MB conda-forge scipy-0.17.0 |np111py35_nomkl_4 29.5 MB setuptools-40.4.3 | py35_0 572 KB conda-forge sip-4.18 | py35_1 466 KB conda-forge six-1.11.0 | py35_1 21 KB conda-forge snowballstemmer-1.9.0 | py_0 51 KB conda-forge sortedcontainers-2.1.0 | py_0 25 KB conda-forge sphinx-2.1.2 | py_0 1.4 MB conda-forge sphinxcontrib-applehelp-1.0.1| py_0 28 KB conda-forge sphinxcontrib-devhelp-1.0.1| py_0 22 KB conda-forge sphinxcontrib-htmlhelp-1.0.2| py_0 27 KB conda-forge sphinxcontrib-jsmath-1.0.1 | py_0 7 KB conda-forge sphinxcontrib-qthelp-1.0.2 | py_0 25 KB conda-forge sphinxcontrib-serializinghtml-1.1.1| py_0 24 KB conda-forge tblib-1.4.0 | py_0 12 KB conda-forge tk-8.6.9 | hed695b0_1002 3.2 MB conda-forge toolz-0.10.0 | py_0 46 KB conda-forge tornado-5.1.1 | py35h470a237_0 671 KB conda-forge urllib3-1.23 | py35_1 154 KB conda-forge wheel-0.32.0 | py35_1000 34 KB conda-forge xz-5.2.4 | h14c3975_1001 366 KB conda-forge yaml-0.1.7 | h14c3975_1001 78 KB conda-forge zict-1.0.0 | py_0 10 KB conda-forge zlib-1.2.11 | h516909a_1005 105 KB conda-forge ------------------------------------------------------------ Total: 256.5 MB

Are you sure? For the minimal or the default build?

Yep, the doc-min-dependencies build. You probably still have the free channel in your .condarc. In your output your scipy and matplotlib do not come from conda-forge (they do not have conda-forge on the right hand side)

Add free conda channel if we are in the doc-min-dependencies build. Also make sure that cython version is greater that the minimum supported version.

lesteve · 2019-07-30T12:16:24Z

I pushed a single commit using CIRCLE_JOB so that we use the conda free channel only in our doc-min-dependencies job. cython>=0.28.5 (our minimum supported cython version) needs to be added to the conda create command otherwise the solver with get cython 0.26 when using the free channel.

I feel this is the simplest way to get CircleCI fixed in master and carry on working on slighly more fun/important things ;-)

rth

LGTM. Actually, @thomasjpfan did a similar solution in #14508

Sorry, I vaguely remembered it but for some reason though that one was the issue and this the PR.

We can merge either one.

glemaitre · 2019-07-30T13:22:31Z

I merged #14508 which introduce the cython version in the config file instead.
Otherwise, the changes were the same. Hope that I did not upset @lesteve :)

Closing then.

lesteve · 2019-07-30T13:24:30Z

Right I did not see #14508, that would have saved me some time ... CIRCLE_JOB seems a bit more robust than PYTHON_VERSION and my comment is a bit more explanatory.

rth · 2019-07-30T13:25:30Z

Maybe re-open and update both?

lesteve · 2019-07-30T13:34:44Z

I opened a new PR at #14519 (I could not reopen this one because I had to rebase on master and force push and apparently you can not reopen a PR if the branch was force pushed, oh well).

rth reviewed Jul 30, 2019

View reviewed changes

lesteve force-pushed the fix-min-dependencies-doc branch from e949a29 to b0b1804 Compare July 30, 2019 11:55

[doc build] Fix min-dependencies doc build.

6a1b58b

Add free conda channel if we are in the doc-min-dependencies build. Also make sure that cython version is greater that the minimum supported version.

lesteve force-pushed the fix-min-dependencies-doc branch from b0b1804 to 6a1b58b Compare July 30, 2019 12:08

rth approved these changes Jul 30, 2019

View reviewed changes

glemaitre closed this Jul 30, 2019

rth mentioned this pull request Jul 30, 2019

[MRG] BLD Adds conda free channel for python 3.5 on circleci #14508

Merged

lesteve mentioned this pull request Jul 30, 2019

[MRG] Minor improvements to build_doc.sh (related to doc-min-dependencies fix) #14519

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[MRG] Fix min-dependencies doc build #14515

[MRG] Fix min-dependencies doc build #14515

Uh oh!

lesteve commented Jul 30, 2019 •

edited

Loading

Uh oh!

rth Jul 30, 2019

Uh oh!

lesteve Jul 30, 2019

Uh oh!

rth Jul 30, 2019

Uh oh!

lesteve Jul 30, 2019 •

edited

Loading

Uh oh!

rth Jul 30, 2019

Uh oh!

lesteve commented Jul 30, 2019 •

edited

Loading

Uh oh!

rth left a comment

Uh oh!

glemaitre commented Jul 30, 2019

Uh oh!

lesteve commented Jul 30, 2019

Uh oh!

rth commented Jul 30, 2019

Uh oh!

lesteve commented Jul 30, 2019

Uh oh!

Uh oh!

Uh oh!

[MRG] Fix min-dependencies doc build #14515

[MRG] Fix min-dependencies doc build #14515

Uh oh!

Conversation

lesteve commented Jul 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rth Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

lesteve Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

rth Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

lesteve Jul 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rth Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

lesteve commented Jul 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rth left a comment

Choose a reason for hiding this comment

Uh oh!

glemaitre commented Jul 30, 2019

Uh oh!

lesteve commented Jul 30, 2019

Uh oh!

rth commented Jul 30, 2019

Uh oh!

lesteve commented Jul 30, 2019

Uh oh!

Uh oh!

lesteve commented Jul 30, 2019 •

edited

Loading

lesteve Jul 30, 2019 •

edited

Loading

lesteve commented Jul 30, 2019 •

edited

Loading