MNT Use conda-forge instead of the main channel in CI #21337

adrinjalali · 2021-10-15T08:45:04Z

Somehow anaconda has patched python in the main channel that it breaks our CI. We should be using conda-forge instead, maintainers don't maintain what goes on the main channel.

Fixes #21324, #21325, #21326, #21327
Closes #21329

adrinjalali · 2021-10-15T08:51:15Z

@glemaitre seems like our CI uses the main channel and not conda-forge, testing conda-forge here. We shouldn't be using the main channel anyway.

ogrisel · 2021-10-15T09:40:02Z

I am fine to only run the docstests on conda-forge CI jobs if that allows to not waste time trying to understand the cause of the scientific notations in the doctests (#21324).

However, it's a problem if we no longer run the scikit-learn tests at all against numpy and scipy from the Anaconda main channel as I suspect a significant fraction of our users still do and problems such as #21326 are more serious than a wrong print format.

adrinjalali · 2021-10-15T09:49:55Z

Anaconda main channel is not maintained by maintainers of the packages, we have pretty much no control over them. I don't want us to spend any energy on a system which is a black box and we don't control. Unless we're paid to maintain against packages in the main channel, I don't think we should be spending time and energy on it.

adrinjalali · 2021-10-15T09:50:22Z

The failing test is a numpydoc validation one: https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=33615&view=logs&j=78a0bf4f-79e5-5387-94ec-13e67d216d6e&t=dbc16e8a-e84c-5cb6-14f6-590d872d6e0b

@glemaitre ideas?

glemaitre · 2021-10-15T09:53:11Z

Regarding the numpy validation, I added the PR of @thomasjpfan but without merging main before merging. In the meanwhile, I assume that we added a function that does not pass the numpydoc validation. @ogrisel is doing a PR.

adrinjalali · 2021-10-15T09:54:06Z

So should we merge this and @ogrisel 's PR pulls this to make the CI green, or the other way around?

glemaitre · 2021-10-15T09:56:32Z

So should we merge this and @ogrisel 's PR pulls this to make the CI green, or the other way around?

Merge @ogrisel PR first, "rebase" this one, and try to understand the segfault:
https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=33615&view=logs&jobId=78a0bf4f-79e5-5387-94ec-13e67d216d6e&j=cdb9ac20-7056-5f9a-0c7e-7c443f61f8a0&t=9e067a35-42f8-5d7a-14d3-926d2bc32e73

This one is fresh and I never seen it before.

adrinjalali · 2021-10-15T09:58:14Z

I think I've seen that crash, and a restart of the CI fixed the issue. But we'll check once Olivier's PR is pulled here.

…to channel-test

adrinjalali · 2021-10-15T12:08:23Z

should we merge this one, w/o scipy1.2 upgrade, and fix that issue separately?

This reverts commit 139fde4.

adrinjalali · 2021-10-15T12:54:53Z

Note that both failures are on py3.7 which we're dropping for the next release

lesteve · 2021-10-15T13:02:32Z

I can reproduce the test_rank_deficient_design failure locally on Python 3.9 see #21340.

ogrisel · 2021-10-15T14:17:41Z

Some of the original issues have gone away magically (probably caused by a transient problem in the conda main channel packages). I am not sure this PR is needed anymore.

adrinjalali · 2021-10-15T14:20:54Z

I would be happier if we actually test against conda-forge, something people can use for free without having to pay, rather than conda/main. I think our duty is to those users more than conda/main users. Happy to have conda/main tests in a cron job to report issues to anaconda if we want, but I think we should be testing against conda-forge.

ogrisel · 2021-10-15T14:46:14Z

We do already test against conda-forge dependencies (and pypi.org dependencies) in other builds.

I think it's good to test against various popular package providers to shake our test assertions and make our test less brittle.

I agree that from time to time this causes our CI to fail for transient reasons outside of our control but I think it's rare enough we can bear it, no?

adrinjalali · 2021-10-15T14:52:32Z

I must be missing something, because I don't see anywhere us checking against conda-forge envs.

jjerphan · 2021-10-15T14:54:09Z

I just have observed this problem with scientific notation for #20254, see this log.

ogrisel · 2021-10-15T16:03:32Z

I just have observed this problem with scientific notation for #20254, see this log.

I re-ran the failed job and it went away. This build probably failed just before the transient problem went away.

ogrisel · 2021-10-16T15:46:09Z

#21324 is happening again...

ogrisel · 2021-10-16T16:05:03Z

Instead of hardcoding conda-forge in the install.sh script, I updated the yaml config. I also took this opportunity to rename the entries of the build matrix to be more consistent and informative.

ogrisel · 2021-10-16T16:50:46Z

The segfault on pylatest_pip_openblas_pandas has already been reported here: #21339.

ogrisel · 2021-10-16T16:51:27Z

Weirdly enough we did not get the #21345 failure on this one...

adrinjalali · 2021-10-16T17:24:30Z

@ogrisel I think your import of sklearn is too early in script here

… env

azure-pipelines.yml

ogrisel · 2021-10-18T10:19:22Z

azure-pipelines.yml

        DISTRIB: 'conda'
+        CONDA_CHANNEL: 'defaults'  # Anaconda main channel


We need to use the anaconda main channel to test against scipy='min' (that is 1.1.0 currently) because the version shipped in conda-forge has a segfault in cdist.

It's probably not worth wasting time debugging or reporting this segfault because this is a really old version of scipy that we probably won't support anymore once we bump the dependency on Python 3.8 for instance.

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

ogrisel

+1 for merge.

adrinjalali · 2021-10-18T10:41:11Z

I'm happy with the changes, thanks @ogrisel .

@glemaitre feel free to merge if you're happy.

…1337) Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Guillaume Lemaitre <g.lemaitre58@gmail.com>

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Guillaume Lemaitre <g.lemaitre58@gmail.com>

…1337) Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Guillaume Lemaitre <g.lemaitre58@gmail.com>

testing channels

809ecfa

adrinjalali added the No Changelog Needed label Oct 15, 2021

github-actions bot added the Build / CI label Oct 15, 2021

adrinjalali added 2 commits October 15, 2021 10:55

move to matplotlib 2.2.3 since 2.2.2 is not available on conda-forge

f427b21

change matplotlib min version to 2.2.3 in docs

1880d0b

adrinjalali changed the title ~~[DEBUG] testing CI conda channels~~ Use conda-forge instead of the main channel in CI Oct 15, 2021

adrinjalali changed the title ~~Use conda-forge instead of the main channel in CI~~ MNT Use conda-forge instead of the main channel in CI Oct 15, 2021

adrinjalali mentioned this pull request Oct 15, 2021

MNT try non-scientific formatting in numpy output #21329

Closed

glemaitre mentioned this pull request Oct 15, 2021

CI Segfault in the testing of the metrics #21339

Closed

adrinjalali added 4 commits October 15, 2021 12:51

testing channels

b79a2c0

move to matplotlib 2.2.3 since 2.2.2 is not available on conda-forge

9f92c32

change matplotlib min version to 2.2.3 in docs

2bfec4e

Merge branch 'channel-test' of github.com:adrinjalali/scikit-learn in…

757541d

…to channel-test

github-actions bot added the module:utils label Oct 15, 2021

test scipy 1.2.0

139fde4

Revert "test scipy 1.2.0"

ec4598c

This reverts commit 139fde4.

ogrisel added 2 commits October 16, 2021 17:48

Merge branch 'main' into channel-test

8cf732c

Use conda-forge on all Azure builds explicitly

c60c083

ogrisel mentioned this pull request Oct 16, 2021

CI Fixes test_rank_deficient_design #21345

Merged

MAINT display more runtime info before launching tests

931acd2

ogrisel added 2 commits October 17, 2021 01:29

Introspect python runtime from outside of the scikit-learn source folder

fd6f0d8

Use a slightly more recent version of threadpoolctl to debug segfault…

9f98d33

… env

glemaitre self-assigned this Oct 18, 2021

glemaitre added 2 commits October 18, 2021 11:02

Merge remote-tracking branch 'origin/main' into pr/adrinjalali/21337

88958da

MNT switch back to main channel for segfault

68b1cd8

ogrisel reviewed Oct 18, 2021

View reviewed changes

azure-pipelines.yml Outdated Show resolved Hide resolved

ogrisel reviewed Oct 18, 2021

View reviewed changes

Apply suggestions from code review

c740e35

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

ogrisel approved these changes Oct 18, 2021

View reviewed changes

ogrisel merged commit 5c1ad3f into scikit-learn:main Oct 18, 2021

adrinjalali deleted the channel-test branch October 18, 2021 12:23

glemaitre mentioned this pull request Oct 23, 2021

Release 1.0.1 #21404

Merged

10 tasks

glemaitre added a commit that referenced this pull request Oct 25, 2021

MNT Use conda-forge instead of the main channel in CI (#21337)

73cd69a

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Guillaume Lemaitre <g.lemaitre58@gmail.com>

lesteve mentioned this pull request Feb 21, 2022

CI: revert LDFLAGS sysroot work-around #22535

Closed

		DISTRIB: 'conda'
		CONDA_CHANNEL: 'defaults' # Anaconda main channel

Uh oh!

MNT Use conda-forge instead of the main channel in CI #21337

MNT Use conda-forge instead of the main channel in CI #21337

Uh oh!

Conversation

adrinjalali commented Oct 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

ogrisel commented Oct 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

glemaitre commented Oct 15, 2021

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

glemaitre commented Oct 15, 2021

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

lesteve commented Oct 15, 2021

Uh oh!

ogrisel commented Oct 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

ogrisel commented Oct 15, 2021

Uh oh!

adrinjalali commented Oct 15, 2021

Uh oh!

jjerphan commented Oct 15, 2021

Uh oh!

ogrisel commented Oct 15, 2021

Uh oh!

ogrisel commented Oct 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ogrisel commented Oct 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ogrisel commented Oct 16, 2021

Uh oh!

ogrisel commented Oct 16, 2021

Uh oh!

adrinjalali commented Oct 16, 2021

Uh oh!

Uh oh!

ogrisel Oct 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

adrinjalali commented Oct 18, 2021

Uh oh!

Uh oh!

adrinjalali commented Oct 15, 2021 •

edited

Loading

ogrisel commented Oct 15, 2021 •

edited

Loading

ogrisel commented Oct 15, 2021 •

edited

Loading

ogrisel commented Oct 16, 2021 •

edited

Loading

ogrisel commented Oct 16, 2021 •

edited

Loading

ogrisel Oct 18, 2021 •

edited

Loading