Skip to content

MAINT Clean-up more unused Makefile targets #29296

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jun 21, 2024

Conversation

lesteve
Copy link
Member

@lesteve lesteve commented Jun 19, 2024

Follow-up of #29170 and preparing for #29012.

In rough order of controversy potential, from from 1 to 10 (from "no brainer"=1 to "super controversial"=10)

  1. make cython controversy rating 1: never heard of it, not sure what it does, probably nobody uses it
  2. make test-* removal controvery rating 1: and similar have been broken for a few years. In particular since the global random seed plugin, you need to install scikit-learn in your environment or use python -m pytest that adds current working directory to sys.path automatically, nobody has ever complained about this it seems until Unable to run code tests, both serial and parallel #29280 so I am guessing this is not used
  3. make doc, make doc-noplot removal controversy rating 2: undocumented so people go to doc and do make html ormake html-noplot
  4. all removal controversy rating 3: I have a slight preference that people be explicit about what they want to do ... the Makefile will go away one day we use spin ENH Add minimal support for spin as a developer tool #29012. If there is a strong support for having the same behaviour as previously (build + test) I can put it back.

Other changes after @adrin's suggestion with higher controversy rating:

  • add dev for dev-meson to have something less generic, controversy rating 3
  • rename setuptools specific targets controversy rating 5: for example clean-meson rather than clean, inplace-setuptools rather than inplace
  • probably other stuff I have forgotten

Copy link

github-actions bot commented Jun 19, 2024

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: f082a64. Link to the linter CI: here


clean:
$(PYTHON) setup.py clean
rm -rf dist

in: inplace # just a shortcut
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be nice to replace this in target with what we do now though. As in, still be able to do make in

Copy link
Member Author

@lesteve lesteve Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be a bit weird that in is not the same as inplace anymore though ... more generally see #29296 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather rename this to dev-setuptools for consistency.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we could do this, although dev-setuptools would be different to dev-meson in the sense that you need make dev-setuptools each time you change a Cython file.

Keeping make inplace-setuptools you could argue that if some people are still using make in, they can still make in<TAB> so slightly less disruptive. But the number of these people is beween 0 and 1 apparently so 🤷.

I am hoping to look at #29012 again so in the medium term it does not matter that much, so I am fine either way ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also fine either way, we should not bother too much since we expect to have less and less people installing using setuptools.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for dev-setuptools. Otherwise it's too confusing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dev-setuptools it is, I also reordered the targets in rough order of importance

@adrinjalali
Copy link
Member

Can we then have a make dev then? not to conflict with in and inplace?

Copy link
Member

@adrinjalali adrinjalali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if CI is green.

@lesteve
Copy link
Member Author

lesteve commented Jun 19, 2024

Seriously now this is the HTTP 403 in fetch_california_housing 😅 I must have angered the CI gods somehow ...

@adrinjalali
Copy link
Member

FWIW, on the fairlearn side, we've constantly been getting HTTP errors from openml (cc @TamaraAtanasoska ). I'm not sure why we haven't been getting those here.

@lesteve
Copy link
Member Author

lesteve commented Jun 19, 2024

I don't want to hear about any more weird problems, I am trying to work here please 😅.

Actually right now, I am feeling like a headless chicken running around randomly and bouncing between different red CIs 🐔 😵‍💫

Copy link
Member

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but we should wait for @jeremiedbb's opinion before merging this as he is the last maintainer to still use this AFAIK ;)


clean:
$(PYTHON) setup.py clean
rm -rf dist

in: inplace # just a shortcut
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather rename this to dev-setuptools for consistency.

Makefile Outdated
Comment on lines 25 to 27
clean-meson:
pip uninstall -y scikit-learn

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not keeping this one ? Also, is it enough ? You don't need a git clean ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, since I need to run the commands manually on windows, I find it nice to just have to look into the Makefile to find them

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can keep it if you want, basically you need this if you switch between setuptools and Meson.

If you only use Meson, you do once make dev-meson and then importing sklearn will recompile as needed at import-time. That means, you never have to think do I need to build or not.

Also I find that since scikit-learn is installed as a package that if I want to remove it, pip uninstall scikit-learn -y is something that most people know how to do.

Copy link
Member Author

@lesteve lesteve Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Meson (or rather ninja) in principle you never need to clean. In the case I wanted to make 100% sure there was not something really fishy happening, I did rm -rf build because that's where pip install -e . will put stuff by default something like build/cp312 for a Python 3.12 install.

Copy link
Member

@ogrisel ogrisel Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this is broken (using meson-python 0.16.0):

$ make dev
...
$ rm -rf build  # alternatively: git clean -xdf
$ python -c "import sklearn; sklearn.show_versions()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1138, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1078, in _find_spec
  File "/Users/ogrisel/miniforge3/envs/dev/lib/python3.11/site-packages/_scikit_learn_editable_loader.py", line 310, in find_spec
    tree = self._rebuild()
           ^^^^^^^^^^^^^^^
  File "/Users/ogrisel/miniforge3/envs/dev/lib/python3.11/site-packages/_scikit_learn_editable_loader.py", line 338, in _rebuild
    if self._work_to_do(env):
       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ogrisel/miniforge3/envs/dev/lib/python3.11/site-packages/_scikit_learn_editable_loader.py", line 325, in _work_to_do
    p = subprocess.run(dry_run_build_cmd, cwd=self._build_path, env=env, capture_output=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ogrisel/miniforge3/envs/dev/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ogrisel/miniforge3/envs/dev/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/ogrisel/miniforge3/envs/dev/lib/python3.11/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/ogrisel/code/scikit-learn/build/cp311'

Ideally one would want meson-python to tolerate that and rebuild the missing build directory, no?

Copy link
Member Author

@lesteve lesteve Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 I probably was not clear in my previous comment before, here is what I wanted to say:

  • if you are only working with Meson, you do make dev-meson once and then use scikit-
    learn since importing will retrigger compilation. I guess people who have switched to Meson probably never had to think much about "cleaning", it just works
  • pip uninstall -y scikit-learn is the way to "clean" Meson, in case you want to switch to setuptools. I am guessing most people don't have to do this unless you switch to an old PR.
  • every once in a while, like every month, I am looking at a fishy behaviour, and just because I am paranoid I do rm -rf build. This is clearly not needed and also not a good idea to do in general because you may have other build there of scikit-learn built for other other environments and Python versions.

All in all if you insist you want a make clean and make clean-meson I would stick with the previous behaviour of only doing pip uninstall -y scikit-learn

If you want an example of a fishy behaviour, I was looking why somehow we lost pytest assertion rewriting in Meson (i.e. pytest is not showing the value of values on each side of the assertion see #29253 (comment)), so I was switching between Meson and setuptools, and I did not want to take any risks so I did rm -rf build.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's helpful to have an official way to handle the cleaning of a dev build that does not result in one of the broken states I posted above.

Because indeed some people are paranoid and will attempt to do a "clean" operation manually from time to time (even if it's useless, just in case) and end up in the same state as I did.

Copy link
Member

@jeremiedbb jeremiedbb Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because indeed some people are paranoid and will attempt to do a "clean" operation manually from time to time

I'm one of those guys and I'm pretty sure it got me out of a very broken state sometimes :)
Anyway, even if it trivial, I'd rather have the shortcut, and I like being able to see all the necessary commands explicitly at one place.

Copy link
Member Author

@lesteve lesteve Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK is my last commit good enough or you want an additional clean-paranoid target that does rm -rf build (on top of clean that does pip uninstall -y scikit-learn 😉)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good enough for me

@jeremiedbb
Copy link
Member

LGTM but we should wait for @jeremiedbb's opinion before merging this as he is the last maintainer to still use this AFAIK ;)

well I just run make dev-meson once :)

@lesteve
Copy link
Member Author

lesteve commented Jun 20, 2024

well I just run make dev-meson once :)

once is better than never 😉

@lesteve lesteve force-pushed the clean-up-makefile-bis branch from 0a624ea to 6e48562 Compare June 21, 2024 14:27
@lesteve
Copy link
Member Author

lesteve commented Jun 21, 2024

Auto-merging my own PR with 3 approvals, I looked at all the comments, and I am reasonably confident they have been addressed.

If I missed something we can tackle this in a follow-up PR.

Summary:

  • for Meson: make dev and make clean (or make dev-meson and make clean-meson). make clean does pip uninstall -y scikit-learn
  • for setuptools: make dev-setuptools and make clean-setuptools
  • make without arguments now shows some simple help message. I (re)discovered that make without arguments actually picks the first rule in the Makefile apparently

@lesteve lesteve enabled auto-merge (squash) June 21, 2024 14:28
@lesteve lesteve merged commit 69b482c into scikit-learn:main Jun 21, 2024
28 checks passed
@lesteve lesteve deleted the clean-up-makefile-bis branch June 21, 2024 15:51
@jeremiedbb jeremiedbb mentioned this pull request Jul 2, 2024
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants