[MRG] MAINT moving to modern way of building docs. #9402

NelleV · 2017-10-14T20:11:52Z

Switching to Makefile to build the documentation like the rest of the Python
ecosystem.
Related to MEP10

PR Summary

PR Checklist

Has Pytest style unit tests
Code is PEP 8 compliant
New features are documented, with examples if plot related
Documentation is sphinx and numpydoc compliant
Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

anntzer · 2017-10-14T20:50:47Z

A recent sphinx will generate a much simpler makefile

# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line.
SPHINXOPTS    =
SPHINXBUILD   = python -msphinx
SPHINXPROJ    = foo
SOURCEDIR     = .
BUILDDIR      = _build

# Put it first so that "make" without argument is like "make help".
help:
	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

I would suggest sticking to it (basically sphinx already provides all the necessary machinery), and changing the CI build to use

make SPHINXOPTS=-W html

(disallow warnings -- when building locally I'd much rather have the build go to the end and give me all warnings)
and likewise document

make SPHINXOPTS=-Dplot_gallery=0 html

(which can be combined more easily with -W)

and add the secondary targets manually to the makefile.
(on Windows -- does anyone build the docs from Windows? set SPHINXOPTS=-W & make should work)

But in general I agree with the idea.

anntzer · 2017-10-14T20:59:09Z

doc/README.txt

@@ -34,9 +34,9 @@ generated.

 You can build the documentation with several options:


options -> targets

choldgraf · 2017-10-14T22:52:58Z

I think this is a great idea...not really any specific complaint from me so long as the docs build fine on the tests/CI. I'm 50/50 on including both a Makefile and a .bat makefile...I know that the .bat is how you'd build the docs on windows, but I can envision someone making changes to one without remembering to change the other...

choldgraf

Approve pending tests/CI pass

anntzer

Feel free to dismiss this if you really don't want to switch to the shorter, newer makefile (but why?).

tacaswell · 2017-10-15T02:22:06Z

doc/make.py

-DEPSY_DEFAULT = "_static/depsy_badge_default.svg"
-
-
-def fetch_depsy_badge():


Can this be ported over as well? This works around depsy not having an https end point which results in matplotlib.org warning about non-secure resources.

I am also ok with just removing the depsy badge, it seems to not have been updated recently.

I'll deal with that once I figure out the examples not running issue.

I'm in favor of removing all the badges that we have. We have a problem of signal to noise ratio on the website, so we need to think about what is necessary and what can be remove, and I think this can safely been remove (basically, those badges are only understood by people who already know this information).

tacaswell

👍 in principle, but I think the depsy badge handling needs to be ported.

tacaswell · 2017-10-15T02:35:43Z

doc/make.py

-DEPSY_DEFAULT = "_static/depsy_badge_default.svg"
-
-
-def fetch_depsy_badge():


I am also ok with just removing the depsy badge, it seems to not have been updated recently.

tacaswell · 2017-10-15T02:48:08Z

doc/README.txt

-* `-n N` enables parallel build of the documentation using N process.
+* `make html-noplot` doesn't save the gallery's images. Allows for fast build.
+* `make html-allow-warnings`: Don't turn Sphinx warnings into errors.
+* `make -j N` enables parallel build of the documentation using N process.


Sorry if this a silly question, but what in the makefile deals with this? It still seems to only use one processor when running locally....

I'd guess the correct invocation is make SPHINXOPTS=-j$N ...

tacaswell · 2017-10-15T02:55:31Z

doc/make.py

-
-# Change directory to the one containing this file
-current_dir = os.getcwd()
-os.chdir(os.path.dirname(os.path.join(current_dir, __file__)))


I suspect that this line is the source of the import failures.

The old make.py compiles fine without those lines, so I don't think that this is the problem.

I've opened a PR on sphinx-gallery that "fixes" the problem. I have no clue how the old make.py managed to build the examples considering the sphinx-gallery code I patched.

I would not be surprised that my patch gets rejected, as it opens the door to examples that are not self contained (which is our problem). I think the correct solution would be to make the units examples self-contained and easier to understand.

tacaswell · 2017-10-15T02:56:09Z

I am also getting local failures

WARNING: /home/tcaswell/source/p/matplotlib/examples/units/radian_demo.py failed to execute correctly: Traceback (most recent call last):                                                                                                                                                                                                                                                                                               
  File "/home/tcaswell/source/p/matplotlib/examples/units/radian_demo.py", line 17, in <module>
    from basic_units import radians, degrees, cos
ModuleNotFoundError: No module named 'basic_units'

WARNING: /home/tcaswell/source/p/matplotlib/examples/units/units_sample.py failed to execute correctly: Traceback (most recent call last):                                                                                                                                                                                                                                                                                              
  File "/home/tcaswell/source/p/matplotlib/examples/units/units_sample.py", line 16, in <module>
    from basic_units import cm, inch
ModuleNotFoundError: No module named 'basic_units'

WARNING: /home/tcaswell/source/p/matplotlib/examples/units/units_scatter.py failed to execute correctly: Traceback (most recent call last):                                                                                                                                                                                                                                                                                             
  File "/home/tcaswell/source/p/matplotlib/examples/units/units_scatter.py", line 15, in <module>
    from basic_units import secs, hertz, minutes
ModuleNotFoundError: No module named 'basic_units'

WARNING: /home/tcaswell/source/p/matplotlib/examples/units/bar_demo2.py failed to execute correctly: Traceback (most recent call last):                                                                                                                                                                                                                                                                                                 
  File "/home/tcaswell/source/p/matplotlib/examples/units/bar_demo2.py", line 17, in <module>
    from basic_units import cm, inch
ModuleNotFoundError: No module named 'basic_units'

WARNING: /home/tcaswell/source/p/matplotlib/examples/units/annotate_with_units.py failed to execute correctly: Traceback (most recent call last):                                                                                                                                                                                                                                                                                       
  File "/home/tcaswell/source/p/matplotlib/examples/units/annotate_with_units.py", line 15, in <module>
    from basic_units import cm
ModuleNotFoundError: No module named 'basic_units'

WARNING: /home/tcaswell/source/p/matplotlib/examples/units/bar_unit_demo.py failed to execute correctly: Traceback (most recent call last):                                                                                                                                                                                                                                                                                             
  File "/home/tcaswell/source/p/matplotlib/examples/units/bar_unit_demo.py", line 15, in <module>
    from basic_units import cm, inch
ModuleNotFoundError: No module named 'basic_units'

WARNING: /home/tcaswell/source/p/matplotlib/examples/units/artist_tests.py failed to execute correctly: Traceback (most recent call last):                                                                                                                                                                                                                                                                                              
  File "/home/tcaswell/source/p/matplotlib/examples/units/artist_tests.py", line 23, in <module>
    from basic_units import cm, inch
ModuleNotFoundError: No module named 'basic_units'

WARNING: /home/tcaswell/source/p/matplotlib/examples/units/ellipse_with_units.py failed to execute correctly: Traceback (most recent call last):                                                                                                                                                                                                                                                                                        
  File "/home/tcaswell/source/p/matplotlib/examples/units/ellipse_with_units.py", line 12, in <module>
    from basic_units import cm
ModuleNotFoundError: No module named 'basic_units'

generating gallery for gallery/units... [100%] basic_uni

NelleV · 2017-10-15T03:03:54Z

@tacaswell I'm not getting local failures. Which version of sphinx & sphinx-gallery are you using? Are you using some form of virtual environments (conda, virtualenv)?

tacaswell · 2017-10-15T03:08:33Z

This is venv

23:07 $ pip list | grep -i sphinx
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
Sphinx (1.6.3)
sphinx-gallery (0.1.13)
sphinx-rtd-theme (0.2.4)
sphinxcontrib-websupport (1.0.1)

I think this is the same issue that we are seeing on circle.

NelleV · 2017-10-15T03:11:02Z

@tacaswell Yes. I'm reproducing the errors locally. Thanks

- Switching to Makefile to build the documentation like the rest of the Python ecosystem. - Related to MEP10 closes #5798

In addition, sphinx 1.6.4 is buggy

NelleV · 2017-10-20T14:36:00Z

This PR is ready to be reviewed.

tacaswell

Left a few questions. Still a bit unclear how -j 4 would make it in for parallel builds.

tacaswell · 2017-10-20T15:10:02Z

doc-requirements.txt

 colorspacious
 ipython
 mock
 numpydoc
 pillow
-sphinx-gallery>=0.1.12
+sphinx-gallery


Why drop the pinning on this?

That was a mistake… putting it back.

tacaswell · 2017-10-20T15:14:21Z

.circleci/config.yml

@@ -28,6 +28,35 @@ apt-run:  &apt-install
      libgeos-dev \
      otf-freefont

+conda36-run:  &conda36-install


Why switch to conda? It looks like all we are using is python and then pip installing everything else.

I switch to conda only for the environment: we now need to access the scripts "sphinx-build", and not only the library as before. When installing with --user, we either need to change the PYTHONPATH manually, or use conda.

Why not just use python -msphinx instead? AFAIK sphinx-build is a strict synonym to it, except that python -msphinx looks up PYTHONPATH (which is the same as what we did before in make.py) whereas sphinx-build uses PATH (which may be set differently).

This relates to my earlier point about using the shorter/newer makefile generated by recent versions of sphinx.

tacaswell · 2017-10-20T15:15:10Z

.circleci/config.yml

+  command: |
+    export PATH="$HOME/miniconda/bin:$PATH"
+    source activate testenv
+    pip install -ve .


do the docs build with a non-editable installation?

They should. I can switch to pip install -v . if you prefer.

Concern about depsy badge addressed.

NelleV · 2017-10-20T15:41:33Z

@tacaswell I'm not sure what you mean by the parallel build. make -j N is how you parallelize with make, and sphinx-build pulls in this information. Here are all the options supported by make (notice the last one :) ):

-b	Builder to use; defaults to html. See the full list of builders above.
-a	Generate output for all files; without this option only output for new and changed files is generated.
-E	Ignore cached files, forces to re-read all source files from disk.
-d	Path to cached files; defaults to /.doctrees.
-j	Build in parallel with N processes where possible.
-c	Locate the conf.py file in the specified path instead of .
-C	Specify that no conf.py file at all is to be used. Configuration can only be set with the -D option.
-D <setting=value>
	Override a setting from the configuration file.
-t	Define tag for use in “only” blocks.
-A <name=value>
	Pass a value into the HTML templates (only for HTML builders).
-n	Run in nit-picky mode, warn about all missing references.
-v	Increase verbosity (can be repeated).
-N	Prevent colored output.
-q	Quiet operation, just print warnings and errors on stderr.
-Q	Very quiet operation, don’t print anything except for errors.
-w	Write warnings and errors into the given file, in addition to stderr.
-W	Turn warnings into errors.
-T	Show full traceback on exception.
-P	Run Pdb on exception.

NelleV · 2017-10-20T15:47:24Z

@tacaswell never mind… I just understood your question.
I'll look into that.

QuLogic

Duplicate templating should be removed.

QuLogic · 2017-10-20T21:27:35Z

.circleci/config.yml

+    conda create -n testenv --yes --quiet python
+    source activate testenv
+
+conda27-run:  &conda27-install


All of these are templated so you don't need to write out two copies for Python 3 and 2. There should not be a separate 36 and 27 template. You can use an environment variable to differentiate if necessary (see the NUMPY variable for example).

choldgraf · 2017-10-21T17:31:27Z

is there a reason this was closed? seems like it is getting close!

NelleV · 2017-10-21T23:40:48Z

I'm not going to have time to fix the current blockers for merge anytime soon.

NelleV added the Documentation label Oct 14, 2017

NelleV requested review from tacaswell and choldgraf October 14, 2017 20:11

anntzer mentioned this pull request Oct 14, 2017

MAINT Documentation on doc is outdated #9403

Merged

6 tasks

anntzer reviewed Oct 14, 2017

View reviewed changes

doc/README.txt

@@ -34,9 +34,9 @@ generated.

You can build the documentation with several options:

Copy link

Contributor

anntzer Oct 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

options -> targets

choldgraf approved these changes Oct 14, 2017

View reviewed changes

anntzer requested changes Oct 14, 2017

View reviewed changes

tacaswell added this to the 2.2 milestone Oct 15, 2017

tacaswell reviewed Oct 15, 2017

View reviewed changes

tacaswell previously requested changes Oct 15, 2017

View reviewed changes

tacaswell reviewed Oct 15, 2017

View reviewed changes

NelleV changed the title ~~MAINT moving to modern way of building docs.~~ [WIP] MAINT moving to modern way of building docs. Oct 15, 2017

NelleV mentioned this pull request Oct 15, 2017

[MRG] FIX import local modules in examples sphinx-gallery/sphinx-gallery#305

Merged

NelleV added 10 commits October 19, 2017 09:27

MAINT moving to modern way of building docs.

cf33384

- Switching to Makefile to build the documentation like the rest of the Python ecosystem. - Related to MEP10 closes #5798

DOC FIX doc build options

68cead4

MAINT update circle-ci to use conda environments

2a7ab00

DOC update the documentation on writing doc

4aaeed3

debugging attempt

11568d4

MAINT circe-ci uses doc-requirements.txt

517de77

In addition, sphinx 1.6.4 is buggy

FIX the doc build directory is 'build' not '_build'

3f9399f

FIX adding current dir to examples that need local imports

2cb396b

MAINT Go look for linkchecker in the correct dir

77846ee

Matplotlib doesn't build doc in html/stable

2ac57d7

MAINT remove the depsy badge

712774f

NelleV changed the title ~~[WIP] MAINT moving to modern way of building docs.~~ [MRG] MAINT moving to modern way of building docs. Oct 20, 2017

tacaswell reviewed Oct 20, 2017

View reviewed changes

QuLogic requested changes Oct 20, 2017

View reviewed changes

NelleV closed this Oct 21, 2017

NelleV deleted the 5798_makefile branch November 7, 2017 20:08

QuLogic modified the milestones: needs sorting, v2.2.0 Feb 12, 2018

		@@ -34,9 +34,9 @@ generated.

		You can build the documentation with several options:

		DEPSY_DEFAULT = "_static/depsy_badge_default.svg"


		def fetch_depsy_badge():

[MRG] MAINT moving to modern way of building docs. #9402

[MRG] MAINT moving to modern way of building docs. #9402

Conversation

NelleV commented Oct 14, 2017

PR Summary

PR Checklist

anntzer commented Oct 14, 2017 • edited Loading

Choose a reason for hiding this comment

choldgraf commented Oct 14, 2017

choldgraf left a comment

Choose a reason for hiding this comment

anntzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tacaswell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tacaswell commented Oct 15, 2017

NelleV commented Oct 15, 2017

tacaswell commented Oct 15, 2017

NelleV commented Oct 15, 2017

NelleV commented Oct 20, 2017

tacaswell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NelleV commented Oct 20, 2017

NelleV commented Oct 20, 2017

QuLogic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

choldgraf commented Oct 21, 2017 • edited Loading

NelleV commented Oct 21, 2017

anntzer commented Oct 14, 2017 •

edited

Loading

choldgraf commented Oct 21, 2017 •

edited

Loading