This PR can be roughly split into two parts. We can discuss which of these parts are desired, and whether we should target main or new_web_theme. For maintainers to get a rough how things would look under pydata-sphinx-theme, here is a temporary build that includes these changes: https://charlie-xiao.github.io/scikit-learn-pst/stable/index.html, and you can check out the API page. codecov is failing because of my newly added script for generating pages, and I did not fix it because it is yet to be determined whether we want to use the script.

• Used a Python script to generate module API pages instead of writing separate ones. This is mainly for consistency in formatting because I found several inconsistencies in the original modules/classes. However, if this is not desired, please let me know so I can revert. This would not be much work: I can just use my current script to generate the pages, remove the gitignore entry and it is done.

• Split the single page modules/classes into a two-level structure, i.e., api/index links to each module, e.g., api/sklearn.model_selection, then we can link from there two the functions and classes in the sklearn.model_selection module. Why this change? In short, this is to better suit pydata-sphinx-theme. In particular,

  • The primary sidebar of the current theme shows the in-page toc, while that of pydata-sphinx-theme is responsible for the documentation structure. To be more specific, with the current modules/classes the primary sidebar will show all functions/classes without a split because those are the pages that are one-level lower than modules/classes in the toctree. One may argue that we can hide the primary sidebar in pydata-sphinx-theme and only use its secondary sidebar that is responsible for the in-page toctree. This does not solve the root problem. Since the primary sidebar is still ill-formatted, all subpages in that same toctree (i.e., classes and functions) must also hide their primary sidebar. This degrades the flexibility in navigating through the API reference.

  • As glemaitre has mentioned in #26809 (comment), we may not want the module names to be repeated over and over again, since they are too long to fit into sidebars and monospace inline code disables word wrap (at least this is the case for pydata-sphinx-theme). In order to have only class/function names without the module name in the front, it is important to have the two-level structure so that one knows which module he/she is under. With the primary sidebar in pydata-sphinx-theme or with its breadcrumbs component this information can be easily accessed if the two-level structure is implemented.

  • There are projects that use a similar two-level structure for their API references:

    • imbalanced-learn: https://imbalanced-learn.org/stable/references/
    • pandas: https://pandas.pydata.org/docs/reference/index.html

Some other things I want to explain:

• Why removing some templates? They are unused, and I think something like the deprecated warning should be handled in the docstrings directly. As for the robots part... emm I'm not sure what it means and whether we are still using it. If so, maybe I shouldn't have removed the templates.

• Why making the functions that generate files directly called when running conf.py instead of registering them to sphinx? First of all, they don't really need the application or environment information, so it is just unnecessary to register them to sphinx. Secondly, autosummary seems to collect stuff very early, and we have to generate the module API pages even before that. I tried registering to different events or adjusting priorities, but there were always some problems, thus this simple yet usable solution.