Skip to content

Circle: Build docs in parallel. #20718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 9, 2022
Merged

Circle: Build docs in parallel. #20718

merged 2 commits into from
Aug 9, 2022

Conversation

QuLogic
Copy link
Member

@QuLogic QuLogic commented Jul 22, 2021

PR Summary

We are supposedly on medium Docker VMs, which offer 2 vCPUs, so we should try to build in parallel.

PR Checklist

  • [n/a] Has pytest style unit tests (and pytest passes).
  • Is Flake 8 compliant (run flake8 on changed files to check).
  • [n/a] New features are documented, with examples if plot related.
  • Documentation is sphinx and numpydoc compliant (the docs should build without error).
  • [n/a] Conforms to Matplotlib style conventions (install flake8-docstrings and run flake8 --docstring-convention=all).
  • [n/a] New features have an entry in doc/users/next_whats_new/ (follow instructions in README.rst there).
  • [n/a] API changes documented in doc/api/next_api_changes/ (follow instructions in README.rst there).

@jklymak
Copy link
Member

jklymak commented Jul 22, 2021

Looks like the build ran out of memory?

@jklymak jklymak marked this pull request as draft July 27, 2021 13:53
@QuLogic
Copy link
Member Author

QuLogic commented Aug 4, 2022 •
edited
Loading

On recent Circle builds, I noticed it said we have access to a higher resource class, so I bumped it up to large, which will hopefully mean it won't run out of RAM this time, and will be a little bit faster with double the vCPU.

QuLogic added 2 commits August 4, 2022 22:53
@QuLogic
Circle: Increase resource class to large
9ac36f3 
This doubles the vCPU and RAM, which will hopefully be enough to run in
parallel.
@QuLogic
Circle: Build docs in parallel.
afef2e6 
We are now on large Docker VMs, which offer 4 vCPUs, so we should try to
build in parallel.
@QuLogic
Copy link
Member Author

QuLogic commented Aug 5, 2022

Re-reading the resource class docs again, it looks like the system really has 32 cores, and since I passed -jauto, it probably was running in 32 processes, which is likely to OOM on any amount of RAM. Testing locally, single-core takes 2.4GB, 2-process takes 2.5GB, 3-process takes 2.65GB, and 4-process takes 3.1GB. The times are 19.6min, 17.1min, 15.7min, and 15.3min, respectively. Unfortunately, sphinx-gallery takes half the time and doesn't run in parallel, so we can't save too much.

But we might as well save the 5 minutes by going to -j4.

@QuLogic QuLogic marked this pull request as ready for review August 5, 2022 04:18
@tacaswell tacaswell added this to the v3.6.0 milestone Aug 9, 2022
@tacaswell tacaswell merged commit d8005dd into matplotlib:main Aug 9, 2022
@QuLogic QuLogic deleted the circle-parallel branch August 9, 2022 19:31
@StefRe StefRe mentioned this pull request Sep 30, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timhoffm timhoffm timhoffm approved these changes

Assignees
No one assigned
Labels
Build Documentation
Projects
None yet
Milestone
v3.6.0
Development

Successfully merging this pull request may close these issues.
4 participants
@QuLogic @jklymak @timhoffm @tacaswell