Hmm, this is close. It works fine when invoked from make html in doc/ but not when run as python setup.py build_sphinx -nW --keep-going from the project root. I must have something broken in specifying the root.

Should be fixed now...

OK, I think the CI failures are expected.

Before I do a bunch of code cleanup, and write more docs (other than the self-documenting of the examples):

1. Does the user API/behaviour for this make sense? In particular: If the user adds an rst file it is put it somewhere in the TOC, versus the gallery adding it to the hidden TOC and perhaps using a no-image thumbnail. The implemented behaviour is my preference, but I don't think it's too difficult to add a thumbnail.
2. Does the general code flow make sense? I found the generation of the gallery, thumbnails, and index pages pretty intertwined with each other, so I may not be doing the alternate routes as cleanly as possible.

Thanks!

So to me this PR is actually two separable things:

1. Allow index.rst as a pass-through/replacement of README.txt
2. Allow *.rst to be copied over

For (1), I like the current design -- it seems simple enough and I don't think it will break things for people. At least I don't think we allow the source and generated dirs to be the same -- if we did, that's a problem for README.txt->index.rst generation, as running make html once will create index.rst in the source dir in that case, thereby making it no longer be regenerated, which is bad.

For (2), I think we might want to think about a proper API. For example, people have asked about being able to copy other stuff (e.g., images) as well (e.g., #1063 (comment)) but my thought so far was "just do this yourself if you want it". But now that it's been brought up at least a couple of times, something like 'copyfile_regex: '' where you can give a non-empty regex of files to copy seems like it would solve both your use case and more general ones (e.g., images within the examples dir) as you could do 'copyfile_regex': r'.*\.rst'.

For (2), I think we might want to think about a proper API. For example, people have asked about being able to copy other stuff (e.g., images) as well (e.g., #1063 (comment)) but my thought so far was "just do this yourself if you want it". But now that it's been brought up at least a couple of times, something like 'copyfile_regex: '' where you can give a non-empty regex of files to copy seems like it would solve both your use case and more general ones (e.g., images within the examples dir) as you could do 'copyfile_regex': r'.*\.rst'.

Hmm, that is probably easy, and may actually simplify my exploratory code above now that I think of it that way. But even simpler, is there any reason to not copy everything in the gallery source directory over to the target directory? Maybe excluding README.*, particularly README.rst?

Yes, because examples can produce/write junk files, people can have crufty .orig files on their systems, etc. What you're talking about would be the equivalent of something like '.*' and I don't think it's a good/safe/backward compatible default

OK, fair. Adding a copyfile_regexp isn't that hard.

Edit: think I did that right, and the code is somewhat simpler as well!

Assuming this is working the way folks are happy with, what is the advice on how to document/test.

Currently I've just added two new galleries: /examples_with_rst and /examples_rst_index. I didn't roll either of these into the main examples because you build with nested_sections = False, so I'd have to put the :toctree for the rst file at the top level. And of course it would not test /examples_rst_index at all, since there is no subsections by index levels if nested_sections=False.

I think there are two options - the first is keep nested_sections=False and add the two new galleries (probably could get away with one, but it's a little clearer as two). The other option is to actually change nested_sections=True which I believe means a little rewriting of the main example Readme, but nothing too major.

Finally, I've not dug into your testing yet, but I assume I can figure it out.

API suggestion:

1. Use wildcard expressions fnmatch instead of regular expressions. These are easier to read and write and very common for files.
2. Use a list instead of a single item.

This reads for example

copy_files = ['document.rst', '*.png']

instead of

copyfile_regex = r'(document\.rst|.*\.png)'

3. (optionally) If you really think regex capability is needed you can support passing a compiled regex: copy_files = ['simple.rst', 're.compile(r'\d{2}\.png')]. But that could always be added later if there is user request.

I'm fine with a list of fnmatch patterns as well if thats what the s-g maintainers prefer.

Looking through our existing config values there are several that use regex and none that use glob/fnmatch, so we should probably stick with regex for within-package consistency

.. this is rebased on #1072

Commit 2 (and 3) are a refactor of how I ignore the README.txt logic if the index.rst exists. I think I prefer the new way, but it's a bit more intertwined with the README.txt logic. The advantage is less duplication. Disadvantage is that the index.rst.new gets made even though it will never get used, which is a bit confusing.

Disadvantage is that the index.rst.new gets made even though it will never get used, which is a bit confusing.

Sounds worth it given the advantages. And it should be plenty fast anyway!

Sounds worth it given the advantages. And it should be plenty fast anyway!

I think so, as the loop I was bailing out of also did a number of other things, like sg_execution_times, and zipped files.

@timhoffm has limited time currently due to family obligations. But happy to wait for him or other folks. I think the API here is relatively opt-in, so it should not affect anyone who doesn't want this pass-through to happen.

Let's try it -- thanks @jklymak !

Thanks for your help and work maintaining sphinx-gallery!