Remove python < 3.5 from CI #12746

adrinjalali · 2018-12-09T12:09:06Z

Fixes #11989
Fixes #12184
Removing python < 3.5 from the CI.

Ubuntu trusty, however has python 3.4 but its support ends on April 2019, which I guess would be our next release?

Should we then focus on what's available on 16.04 instead, and move our min supported versions for other packages as well?

#### Reference Issues/PRs Fixes scikit-learn#11989 #### What does this implement/fix? Removes python versions 2/3.4 and lower from CI config files.

…on2-ci

adrinjalali · 2018-12-09T15:29:34Z

I'm not sure why travis fails on DISTIB="ubuntu". The setup seems to fail to find atlas (not sure if it's related), and fails with

cc1: some warnings being treated as errors
error: Command "/usr/lib/ccache/gcc -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes 
-g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time 
-D_FORTIFY_SOURCE=2 -fPIC -I/usr/lib/python3/dist-packages/numpy/core/include 
-I/usr/lib/python3/dist-packages/numpy/core/include -I/usr/include/python3.5m 
-I/home/travis/build/scikit-learn/scikit-learn/testvenv/include/python3.5m 
-I/usr/include/python3.5m -I/home/travis/build/scikit-learn/scikit-learn/testvenv/
include/python3.5m -c sklearn/neighbors/quad_tree.c -o build/temp.linux-x86_64-3.5/
sklearn/neighbors/quad_tree.o" failed with exit status 1

I must be missing something trivial I guess...

qinhanmin2014 · 2018-12-09T15:47:33Z

For appveyor, please refer to #11582.
I don't think we need this before 0.20.2.

adrinjalali · 2018-12-09T15:55:37Z

I agree, but I guess 0.20.2 is gonna be in a few days?

eamanu

So, from now sklearn does not support python 2?

jnothman · 2018-12-10T10:19:03Z

So, from now sklearn does not support python 2?

From 0.21, which master is tracking the development of.

jnothman · 2018-12-10T10:21:36Z

Travis is unhappy

adrinjalali · 2018-12-10T10:25:18Z

@jnothman , yeah I know, I was hoping one of you would have an easy answer to the error I'm getting there. Otherwise I'd have to launch a VM and check the error on the same environment. Right now I don't have an easy answer to the issue.

jnothman · 2018-12-10T10:47:09Z

Have you tried increasing the cython version, e.g. latest, if only just to test?

adrinjalali · 2018-12-10T13:21:55Z

Adding joblib to the ubuntu test, and using the latest cython, make travis happy. But codecov complains since the parts of the code specific to old packages are not used anymore. For instance, in preprocessing/data.py, it complains about:

if LooseVersion(np.__version__) < '1.9':
    references = references.tolist()

There are a few questions:

Is moving the min-supported versions to Ubuntu Xenial fine (it's the oldest Ubuntu supporting Python 3.5, and from April it will be the oldest supported Ubuntu).
What should be the minimum Cython version?
Should this PR handle removal of code pieces such as the one mentioned above?
Should this PR handle changes in other places such as setup.py? (changing constants such as SCIPY_MIN_VERSION for instace).

Anything else I'm missing?

ogrisel · 2018-12-10T13:37:39Z

Is moving the min-supported versions to Ubuntu Xenial fine (it's the oldest Ubuntu supporting Python 3.5, and from April it will be the oldest supported Ubuntu).

+1

What should be the minimum Cython version?

Would the version from xenial be recent enough?

Should this PR handle removal of code pieces such as the one mentioned above?

+1

Should this PR handle changes in other places such as setup.py? (changing constants such as SCIPY_MIN_VERSION for instance).

Yes and the documentation should also be updated to reflect what are the minimum version supported. In particular the windows builds instructions can be simplified.

adrinjalali · 2018-12-10T13:59:46Z

Would the version from xenial be recent enough?

Unfortunately the build fails with even a minor version newer than what's provided in Xenial. I can kinda brute force and find the minimum cython which works with Xenial, but I'd personally rather use a new (or new-ish) cython, since we can expect developers and maintainers to have a recent cython (I suppose).

ogrisel · 2018-12-10T14:04:57Z

I can kinda brute force and find the minimum cython which works with Xenial, but I'd personally rather use a new (or new-ish) cython, since we can expect developers and maintainers to have a recent cython (I suppose).

Still it would be good to approximately know what is the oldest version of Cython that works for scikit-learn. Maybe try 2 manual iterations of bisect to have rough idea and use that for this PR.

jeremiedbb · 2018-12-10T15:06:50Z

cython <= 0.27 won't work with python 3.7 for sure

adrinjalali · 2018-12-10T15:08:00Z

@jeremiedbb thanks, good to know 👍

qinhanmin2014 · 2018-12-10T15:17:03Z

Maybe we need another job to take care of min dependency in Circle CI?
See #12184

ogrisel

LGTM.

rth

The 0.20.X branch lives its own life independently of master at this point. We can drop CI support for Python 2.7 and 3.4 in master while keeping such CI configuration in the 0.20.X branch for the future 0.20.2, 0.20.3...

I was assuming 0.20.2 would be very soon in which case keeping py2 for a bit longer would simplify backports. But you are right, I suppose we cannot delay it indefinitely. LGTM as well.

rth · 2018-12-13T21:09:23Z

.circleci/config.yml

@@ -1,13 +1,18 @@
 version: 2

 jobs:
-  python3:
+  python3-min-dependency-versions:


Maybe call this, doc-min-dependencies as it shows in CI at each PR and the current name is a bit long. Everything is python3 now so putting it in the title no longer brings useful information..

So, doc-min-dependencies and doc?

ogrisel · 2018-12-14T08:28:35Z

For some reason the LGTM C++ code analysis fails by trying to import sklearn without building it first. This happens when apt is installing the python-typing package:

[2018-12-13 21:57:39] [build] Selecting previously unselected package python-typing. [2018-12-13 21:57:39] [build] (Reading database ... 68239 files and directories currently installed.) [2018-12-13 21:57:39] [build] Preparing to unpack .../python-typing_3.6.2-1_all.deb ... [2018-12-13 21:57:39] [build] Unpacking python-typing (3.6.2-1) ... [2018-12-13 21:57:40] [build] Setting up python-typing (3.6.2-1) ... [2018-12-13 21:57:41] [build] deptrace-server: Package installer exited with code 0 [2018-12-13 21:57:41] [build] Traceback (most recent call last): [2018-12-13 21:57:41] [build] File "setup.py", line 36, in <module> [2018-12-13 21:57:41] [build] import sklearn [2018-12-13 21:57:41] [build] File "/opt/src/sklearn/__init__.py", line 63, in <module> [2018-12-13 21:57:41] [build] from . import __check_build [2018-12-13 21:57:41] [build] File "/opt/src/sklearn/__check_build/__init__.py", line 46, in <module> [2018-12-13 21:57:41] [build] raise_build_error(e) [2018-12-13 21:57:41] [build] File "/opt/src/sklearn/__check_build/__init__.py", line 41, in raise_build_error [2018-12-13 21:57:41] [build] %s""" % (e, local_dir, ''.join(dir_content).strip(), msg)) [2018-12-13 21:57:41] [build] ImportError: No module named _check_build [2018-12-13 21:57:41] [build] ___________________________________________________________________________ [2018-12-13 21:57:41] [build] Contents of /opt/src/sklearn/__check_build: [2018-12-13 21:57:41] [build] setup.py __init__.py __init__.pyc [2018-12-13 21:57:41] [build] _check_build.pyx [2018-12-13 21:57:41] [build] ___________________________________________________________________________ [2018-12-13 21:57:41] [build] It seems that scikit-learn has not been built correctly.

adrinjalali · 2018-12-14T09:00:52Z

That's really odd, I only changed circle-ci job names in the last commit!

rth · 2018-12-14T09:07:37Z

I wouldn't worry too much about LGTM.com it tends to fail at the analysis step sometimes...

ogrisel · 2018-12-14T09:53:03Z

Ok let's merge before we get conflicts then.

ogrisel · 2018-12-14T09:55:38Z

@adrinjalali would you mind doing a new PR to update at least the README and maybe the build / installation doc to remove references to now unsupported python / numpy / scipy versions?

adrinjalali · 2018-12-14T09:58:21Z

Will do my best (I'm still discovering new places with related documentation that I didn't know they're there).

ogrisel · 2018-12-14T10:01:01Z

The following yields some false positives but not too many:

git grep "\b2\.7\b" | grep -v ".csv" | grep -v "externals"

ogrisel · 2018-12-14T10:02:08Z

$ git grep "\b2\.7\b" | grep -v ".csv" | grep -v "externals"
README.rst:.. |Python27| image:: https://img.shields.io/badge/python-2.7-blue.svg
README.rst:- Python (>= 2.7 or >= 3.4)
build_tools/windows/windows_testing_downloader.ps1:    # Function returns a dictionary of packages to download for Python 2.7.
build_tools/windows/windows_testing_downloader.ps1:        "python" = "https://www.python.org/ftp/python/2.7.7/python-2.7.7.msi"
build_tools/windows/windows_testing_downloader.ps1:        "2.7" = Python27URLs
doc/conf.py:    mathjax_path = ('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/'
doc/developers/contributing.rst:All scikit-learn code should work unchanged in both Python 2.7 and 3.4 or
doc/developers/contributing.rst:to code and it certainly requires testing on both 2.7 and 3.4 or newer.
doc/developers/performance.rst:     185        48          130      2.7      0.0              break
doc/index.rst:                    <li><strong>Scikit-learn 0.21 will drop support for Python 2.7 and Python 3.4.</strong>
doc/install.rst:- Python (>= 2.7 or >= 3.4),
doc/install.rst:    Scikit-learn 0.20 is the last version to support Python 2.7 and Python 3.4.
doc/whats_new/v0.18.rst:    Later versions of scikit-learn will require Python 2.7 or above.
doc/whats_new/v0.18.rst:    Later versions of scikit-learn will require Python 2.7 or above.
doc/whats_new/v0.20.rst:  :class:`neighbors.KDTree` and :class:`neighbors.BallTree` in Python 2.7 to
doc/whats_new/v0.20.rst:    Version 0.20 is the last version of scikit-learn to support Python 2.7 and Python 3.4.
examples/bicluster/plot_bicluster_newsgroups.py:    Like Counter.most_common in Python >=2.7.
examples/cluster/plot_feature_agglomeration_vs_univariate_selection.py:plt.figure(figsize=(7.3, 2.7))
examples/svm/plot_svm_kernels.py:          (0, -2.7),
sklearn/utils/testing.py:    Copied from Python 2.7.5 and modified as required.

adrinjalali · 2018-12-14T10:06:49Z

@ogrisel awesome, thanks :)

qinhanmin2014 · 2018-12-14T13:02:09Z

Hmm, Circle CI isn't running on master?

qinhanmin2014 · 2018-12-14T13:13:42Z

Honestly I'd argue that this might make 0.20.2 much difficult to release. Since we've merged this, maybe we should release 0.20.2 ASAP.

This reverts commit 3311e81.

abgrobbel and others added 8 commits September 4, 2018 00:14

[MRG] Remove Python < 3.5 from continuous integration

8f2febf

#### Reference Issues/PRs Fixes scikit-learn#11989 #### What does this implement/fix? Removes python versions 2/3.4 and lower from CI config files.

merge

91026e3

Merge remote-tracking branch 'upstream/master' into maint/remove-pyth…

fc3345e

…on2-ci

fix circleci merge issues

345aedc

add min-required versions

9c6c673

make ubuntu use python3, remove python3.4 from conda scripts

4cd0788

change to xenial dist on travis

21168d0

libatlas-base-dev may be required?

bd1d1ad

test python3.5 on win 32

7b7aeda

eamanu approved these changes Dec 9, 2018

View reviewed changes

adrinjalali added 2 commits December 10, 2018 12:48

try the latest cython on old ubuntu

f57d73e

add joblib to ubuntu instance

84f6d96

test cython 0.26.1

d2eb633

adrinjalali added 2 commits December 10, 2018 16:08

test cython 0.27.3

05ebd29

use cython 0.28.6 (0.28 is the first to support Python3.7)

e2d6495

update setup.py

dd9cf94

ogrisel approved these changes Dec 13, 2018

View reviewed changes

rth approved these changes Dec 13, 2018

View reviewed changes

change circle-ci job names

3e1e8b7

ogrisel merged commit 2bd87f6 into scikit-learn:master Dec 14, 2018

adrinjalali mentioned this pull request Dec 14, 2018

DOC update min required versions in some remaining docs #12778

Merged

This was referenced Dec 14, 2018

MRG Drop legacy python / remove six dependencies #12639

Merged

Remove "from __future__ import XXX" #12779

Closed

adrinjalali mentioned this pull request Dec 14, 2018

MAINT hotfix circle-ci job dependency #12780

Merged

adrinjalali deleted the maint/remove-python2-ci branch December 14, 2018 13:11

qinhanmin2014 mentioned this pull request Dec 14, 2018

MNT Don't persist doc in doc-min-dependencies job #12782

Merged

jeremiedbb mentioned this pull request Dec 20, 2018

[MRG] Use Scipy cython BLAS API instead of bundled CBLAS #12732

Merged

adrinjalali mentioned this pull request Jan 5, 2019

API Deprecate externals.six #12916

Merged

adrinjalali added a commit to adrinjalali/scikit-learn that referenced this pull request Jan 7, 2019

Remove python < 3.5 from CI (scikit-learn#12746)

481666e

qinhanmin2014 mentioned this pull request Apr 3, 2019

MNT Delete _scipy_sparse_lsqr_backport.py #13569

Merged

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Remove python < 3.5 from CI (scikit-learn#12746)

3311e81

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Revert "Remove python < 3.5 from CI (scikit-learn#12746)"

ec02914

This reverts commit 3311e81.

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Revert "Remove python < 3.5 from CI (scikit-learn#12746)"

ab253e1

This reverts commit 3311e81.

koenvandevelde pushed a commit to koenvandevelde/scikit-learn that referenced this pull request Jul 12, 2019

Remove python < 3.5 from CI (scikit-learn#12746)

ddea0d9

albertcthomas mentioned this pull request Feb 14, 2021

Fixing CI paris-saclay-cds/ramp-workflow#262

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove python < 3.5 from CI #12746

Remove python < 3.5 from CI #12746

adrinjalali commented Dec 9, 2018 •

edited

Loading

adrinjalali commented Dec 9, 2018

qinhanmin2014 commented Dec 9, 2018

adrinjalali commented Dec 9, 2018

eamanu left a comment

jnothman commented Dec 10, 2018

jnothman commented Dec 10, 2018

adrinjalali commented Dec 10, 2018

jnothman commented Dec 10, 2018

adrinjalali commented Dec 10, 2018

ogrisel commented Dec 10, 2018

adrinjalali commented Dec 10, 2018

ogrisel commented Dec 10, 2018 •

edited

Loading

jeremiedbb commented Dec 10, 2018

adrinjalali commented Dec 10, 2018

qinhanmin2014 commented Dec 10, 2018

ogrisel left a comment

rth left a comment

rth Dec 13, 2018

adrinjalali Dec 13, 2018

ogrisel commented Dec 14, 2018

adrinjalali commented Dec 14, 2018

rth commented Dec 14, 2018

ogrisel commented Dec 14, 2018

ogrisel commented Dec 14, 2018

adrinjalali commented Dec 14, 2018

ogrisel commented Dec 14, 2018

ogrisel commented Dec 14, 2018

adrinjalali commented Dec 14, 2018

qinhanmin2014 commented Dec 14, 2018

qinhanmin2014 commented Dec 14, 2018

Remove python < 3.5 from CI #12746

Remove python < 3.5 from CI #12746

Conversation

adrinjalali commented Dec 9, 2018 • edited Loading

adrinjalali commented Dec 9, 2018

qinhanmin2014 commented Dec 9, 2018

adrinjalali commented Dec 9, 2018

eamanu left a comment

Choose a reason for hiding this comment

jnothman commented Dec 10, 2018

jnothman commented Dec 10, 2018

adrinjalali commented Dec 10, 2018

jnothman commented Dec 10, 2018

adrinjalali commented Dec 10, 2018

ogrisel commented Dec 10, 2018

adrinjalali commented Dec 10, 2018

ogrisel commented Dec 10, 2018 • edited Loading

jeremiedbb commented Dec 10, 2018

adrinjalali commented Dec 10, 2018

qinhanmin2014 commented Dec 10, 2018

ogrisel left a comment

Choose a reason for hiding this comment

rth left a comment

Choose a reason for hiding this comment

rth Dec 13, 2018

Choose a reason for hiding this comment

adrinjalali Dec 13, 2018

Choose a reason for hiding this comment

ogrisel commented Dec 14, 2018

adrinjalali commented Dec 14, 2018

rth commented Dec 14, 2018

ogrisel commented Dec 14, 2018

ogrisel commented Dec 14, 2018

adrinjalali commented Dec 14, 2018

ogrisel commented Dec 14, 2018

ogrisel commented Dec 14, 2018

adrinjalali commented Dec 14, 2018

qinhanmin2014 commented Dec 14, 2018

qinhanmin2014 commented Dec 14, 2018

adrinjalali commented Dec 9, 2018 •

edited

Loading

ogrisel commented Dec 10, 2018 •

edited

Loading