✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: fae750d. Link to the linter CI: here}

My review arrived too late. It would be good to insert a comment to explain the Why of such config files.

Thanks @ogrisel for the comment, I did not know that robots.txt can have comments. I will have a follow up for that.

I did not know either but it looked the intuitive thing to do and I checked:

https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt#create_rules

I opened #28378 for this.

Reading https://github.com/scikit-learn/scikit-learn/blob/main/build_tools/circle/push_doc.sh more closely to understand how the docs repo works I think the result of this PR will be a robots.txt in https://github.com/scikit-learn/scikit-learn.github.io/tree/main/dev (/dev/ subdirectory). This means we need someone with more experience to chime in.

Are the files above /dev/ created by a script or by hand in https://github.com/scikit-learn/scikit-learn.github.io/tree/main ? The other files there (like https://github.com/scikit-learn/scikit-learn.github.io/blob/main/index.html) have existed unchanged for so long that we can't see the history of how they got created :-/

Maybe @thomasjpfan knows more?

As an alternative adding a robots.txt directly into https://github.com/scikit-learn/scikit-learn.github.io may be simpler ...

That would mean doing a PR on the scikit-learn.github.io repo (better for remembering why we did this) or pushing directly into the repo if we want a quick and dirty thing.

There are precedents of doing things directly in the scikit-learn.github.io repo, see https://github.com/scikit-learn/scikit-learn.github.io/pulls?q=is%3Apr+sort%3Aupdated-desc+is%3Amerged

I am +1 for directly adding into scikit-learn.github.io. Modifying the workflow is feasible, but since this robots.txt does not rely on any variable (such as $dir) there seems to be no reason to use a worflow.

There are precedents of doing things directly in the scikit-learn.github.io repo

Yep and (though I'm not one) I think maintainers need to make some modifications directly in that repo per major/minor release: See the 9th step of https://scikit-learn.org/dev/developers/maintainer.html#making-a-release.

I opened scikit-learn/scikit-learn.github.io#21 to add directly a robots.txt in the .github.io repo.