Hey @rth @thomasjpfan!

Do you mind to review this PR (the log confirms that the ZIP of the HTML is properly uploaded and this link provides the downloadable file?

Nice, thanks @alfaro96 . Since we're not gonna have a ton of these files laying around, I don't think we'd need to worry about the size too much, since it requires cpu cycles to compress them.

Since we're not gonna have a ton of these files laying around, I don't think we'd need to worry about the size too much, since it requires cpu cycles to compress them.

Generally we have an issue with our deployment docs repo size right now: I tried to clone it and at 6% it was at 3.7GB. Because we store the all files for each commit. Even with a clone at --depth 1 the size of the repo (uncompressed) is 2.0GB. I'm kind of surprised we have been allowed by Github to do it, as it's way more than they ToS https://help.github.com/en/github/working-with-github-pages/about-github-pages#usage-limits

So I think anything we can do to reduce the size of files we put there could be worth doing, even at the cost of some CPU time. Though maybe compressing files in previous releases could be a start. The PDF was 48MB in size the ZIP of the PDF is 75MB, so not that different but still a bit more.

Since we're not gonna have a ton of these files laying around, I don't think we'd need to worry about the size too much, since it requires cpu cycles to compress them.

Generally we have an issue with our deployment docs repo size right now: I tried to clone it and at 6% it was at 3.7GB. Because we store the all files for each commit. Even with a clone at --depth 1 the size of the repo (uncompressed) is 2.0GB. I'm kind of surprised we have been allowed by Github to do it, as it's way more than they ToS https://help.github.com/en/github/working-with-github-pages/about-github-pages#usage-limits

So I think anything we can do to reduce the size of files we put there could be worth doing, even at the cost of some CPU time. Though maybe compressing files in previous releases could be a start. The PDF was 48MB in size the ZIP of the PDF is 75MB, so not that different but still a bit more.

The actual size of the ZIP file is 75MB (1.56 times the size of the PDF file). Running optipng under the _images directory with *.png reduces the size of the ZIP file to 59MB (1.23 times the size of the PDF file).

If repository size is a problem, I think that compressing the images with optipng could be interesting, at the cost of slightly increasing the CPU time.

I wonder how much running optipng reduces the size of the zip file...

According to my calculations, running optipng reduces the size of the ZIP file by a 27.11%.

According to my calculations, running optipng reduces the size of the ZIP file by a 27.11%.

Great. How long does it take?

We should probably optimize png and create this Zip in the deploy job not in the job that is building docs (as it runs only for commits on master and only once per commit, unlike the build docs job).

Edit: opened #17568 for additional things we could try.

According to my calculations, running optipng reduces the size of the ZIP file by a 27.11%.

Great. How long does it take?

Compressing the images takes 6:02 (using a docker container with the same hardware resources than CircleCI). I think that is quite affordable.

We should probably optimize png and create this Zip in the deploy job not in the job that is building docs (as it runs only for commits on master and only once per commit, unlike the build docs job).

IIUC, I think that we can compress the images in the dist rule of the Makefile since the following lines

are executed under the same condition than the deploy job

Edit: opened #17568 for additional things we could try.

I have committed these changes to optimize the images (_images/*.png).

Thanks @rth!

Generally we have an issue with our deployment docs repo size right now: I tried to clone it and at 6% it was at 3.7GB. Because we store the all files for each commit. Even with a clone at --depth 1 the size of the repo (uncompressed) is 2.0GB. I'm kind of surprised we have been allowed by Github to do it, as it's way more than they ToS help.github.com/en/github/working-with-github-pages/about-github-pages#usage-limits

Would it be appropriate to use git lfs to at least keep the git out of the server?

Or should we look into less append-only usage of the repo?

Would it be appropriate to use git lfs to at least keep the git out of the server?

git lfs doesn't seem to work with github pages git-lfs/git-lfs#791

Should we move on with this PR? (#18151 solves the case insensitive filesystems)

Should we move on with this PR? (#18151 solves the case insensitive filesystems)

Okay, lets try to move this forward.

The zipped file ends up to have a nested folder structure: html/stable/* where the website is two levels deep. I would expect index.html to be in top directory without any nesting.

For the future, we can work around the github pages being huge as follows:

1. Create another append only repo called scikit-learn-website-backup that is not hosted on github pages.
2. When we deploy here, we push to scikit-learn-website-backup (we can optionally setup git-lfs here).
3. We have a github action in scikit-learn-website-backup to do a force push to scikit-learn.github.io and we keep scikit-learn.github.io to one commit.

This way we have a backup scikit-learn-website-backup and have a github pages that is not huge.

I suspect this is what matplotlib does with its devdocs

I have merged with master to check for possible errors.

Since there aren't any HTML versions for the older versions, the links for the older PDF versions are not generated by build_tools/circle/list_versions.py. I think we should continue to link to the PDF versions for <=0.24 and then have the HTML versions linked for >=1.0.0. (Unfortunately, this means having two code paths in list_versions.py depending on the version.

You are absolutely right. I think that the current solution fixes the problem.

The generated doc/versions.rst file looks good:

:orphan:

Available documentation for Scikit-learn
========================================

Web-based documentation is available for versions listed below:

* `Scikit-learn 1.0.dev0 (dev) documentation <https://scikit-learn.org/dev/>`_
* `Scikit-learn 0.24.0 (stable) documentation <https://scikit-learn.org/stable/>`_ (`PDF 53.7 MB <https://scikit-learn.org/stable//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.23.2 documentation <https://scikit-learn.org/0.23/>`_ (`PDF 51.0 MB <https://scikit-learn.org/0.23//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.22.2 documentation <https://scikit-learn.org/0.22/>`_ (`PDF 48.5 MB <https://scikit-learn.org/0.22//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.21.3 documentation <https://scikit-learn.org/0.21/>`_ (`PDF 46.7 MB <https://scikit-learn.org/0.21//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.20.4 documentation <https://scikit-learn.org/0.20/>`_ (`PDF 45.2 MB <https://scikit-learn.org/0.20//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.19.2 documentation <https://scikit-learn.org/0.19/>`_ (`PDF 42.2 MB <https://scikit-learn.org/0.19//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.18.2 documentation <https://scikit-learn.org/0.18/>`_ (`PDF 46.5 MB <https://scikit-learn.org/0.18//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.17.1 documentation <https://scikit-learn.org/0.17/>`_ (`PDF 46.0 MB <https://scikit-learn.org/0.17//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.16.1 documentation <https://scikit-learn.org/0.16/>`_ (`PDF 56.8 MB <https://scikit-learn.org/0.16//_downloads/scikit-learn-docs.pdf>`_)
* `Scikit-learn 0.15-git documentation <https://scikit-learn.org/0.15/>`_

Note that the ZIP file is not yet available in the website.