Currently build dependencies are manually specified in CI

That's not ideal, because among other things numpy 1.13.3 does not support python 3.8 yet we are building wheels with it as far as I can tell.

Instead I think we should use the build dependencies from https://github.com/scikit-learn/scikit-learn/blob/master/pyproject.toml to install the correct numpy version depending on the Python version, likely with,

pip wheel . -w dist/

instead of

python setup.py sdist_wheel