✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 2857e8b. Link to the linter CI: here}

Reading some doc about robot.txt, it is stated that it's for controlling exploration but not indexation of the website. To control indexation it's advised to use an html tag which is a lot more complex. So let's first try this and see if it's good enough for us before considering more advanced solutions.

Hmmm thinking about it a bit more, I think robots.txt is going to end up in scikit-learn.org/dev rather than at the root. Not sure this will work then ...

Maybe I can do a PR to the https://github.com/scikit-learn/scikit-learn.github.io repo adding robots.txt at the root? We can also revert this PR ...

About the approach, I agree with you that it seems the simplest thing to try. Since ReadTheDocs is recommending using robots.txt I guess this should work, we will see.

About robots.txt not being the "right" way, indeed this is what #8958 (comment) was pointing at, but indeed it sounds more complex. We are also using rel=canonical which should help #8958 (comment) but I think we are using it only in some places (not 100% sure). Also rel=canonical does not work for documentation which have been renamed or have disappeared.

Hmmm thinking about it a bit more, I think robots.txt is going to end up in scikit-learn.org/dev rather than at the root. Not sure this will work then ...

So backporting in 1.6.X (see #30686) will put it in scikit-learn.org/stable ? Then we probably need to do it directly on the webiste repo. And actually we already have a robot.txt there :)

Then we probably need to do it directly on the webiste repo. And actually we already have a robot.txt there :)

Yep, I opened scikit-learn/scikit-learn.github.io#22 on the website repo.

Let's revert this one

I opened #30687 to revert the robots.txt addition to the scikit-learn/scikit-learn repo.

Thanks heaps!!