[WIP] rustdoc testing: Further improve chapters and sections #2298

fmease · 2025-03-20T08:14:10Z

Follow-up to #2295 (applies my remaining review comments).

This is WIP / a draft because I g2g. Will impl all FIXME(fmease)s when I come back.

jieyouxu · 2025-03-20T08:24:43Z

src/rustdoc-internals/rustdoc-test-suite.md

+## Htmldocck-Specific Directives

-In addition to the directives listed here,
-`rustdoc` tests also support most
-[compiletest directives](../tests/directives.html).
+Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments.


Suggestion: even though it's an implementation detail, I think it's worth hammering in the distinction that htmldocck directives are handled by htmldocck and not compiletest.

jieyouxu · 2025-03-20T08:25:36Z

src/rustdoc-internals/rustdoc-test-suite.md

@@ -34,33 +32,41 @@ In this case, the start of the next line should be `//`, with no `@`.

 For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root.

+<!-- FIXME(fmease): Mention that the  regexes match case-sensitively and in single-line mode? -->


Discussion: may also be worth mentioning if the regexes have any string and/or regex escape rules. For instance, I know some compiletest regex directives don't handle this very well.

jieyouxu · 2025-03-20T08:26:59Z

src/rustdoc-internals/rustdoc-test-suite.md

-htmldocck supports the `--bless` option to accept the current subtree
+Htmldocck supports the `--bless` option to accept the current subtree


Question: IIRC the --bless flow requires html-tidy to be installed and available somewhere. Not sure if this is mentioned already. Maybe worth explicitly remarking or linking to that?

jieyouxu · 2025-03-20T08:28:59Z

src/rustdoc-internals/rustdoc-test-suite.md

+<!-- FIXME(fmease):
+Should definitely also mention `//@ aux-crate` and `//@ proc-macro`
+UNLESS we nuke this paragraph entirely and refer to the compiletest section(s)?
+-->


Remark (not-a-review-comment): I think this is where I would like compiletest to support some kind of directive handling "plugin" and allow htmldocck to register plugins. Then we could phrase this as "htmldocck registers additional directive handling to compiletest" either here, or even put this bit under main compiletest chapter.

jieyouxu · 2025-03-20T08:33:49Z

src/rustdoc-internals/rustdoc-test-suite.md

Suggestion: may or may not be helpful for someone looking to adjust how tests/rustdoc works / change htmldocck/compiletest for tests/rustdoc purposes... but here's a very coarse TL;DR on the test infra bits involved on running tests/rustdoc:

Test infra setup

The test infra setup for tests/rustdoc involves the following bits:

Bootstrap (the build system) and its test step that builds:

A suitably-staged sysroot (including the compiler and standard library).

A rustdoc built with the sysroot.

compiletest itself, and running it with the suitable configuration under the rustdoc test mode.

The implementation can be traced back from https://github.com/rust-lang/rust/blob/4e2b096ed6c8a1400624a54f6c4fd0c0ce48a579/src/bootstrap/src/core/build_steps/test.rs#L1349-L1355.

compiletest handles the test discovery, compiletest directive parsing/handling (not to be confused with htmldocck directives) and test running via libtest. See src/tools/compiletest/src/runtest/rustdoc.rs.

compiletest invokes src/etc/htmldocck.py against the test file.

fmease · 2025-06-02T17:37:54Z

src/tests/directives.md


 <!--
 **FIXME(rustdoc)**: what does `check-test-line-numbers-match` do?
 Asked in
 <https://rust-lang.zulipchat.com/#narrow/stream/266220-t-rustdoc/topic/What.20is.20the.20.60check-test-line-numbers-match.60.20directive.3F>.
 -->

+#### Test-suite-specific directives


I've added this subsection because you may very likely land in section Compiletest directives § Rustdoc (the parent section) via fuzzy searching (I noticed that earlier) when (from the perspective of a new user) all you wanted was to get to know what on earth those tool-specific directives meant.

On master, this section wouldn't help you in the slightest.

Re. my change: I wouldn't classify this as duplicating information (which master rustdoc docs do a bunch and it's maddening!) but as necessary redundancy. It's also helpful IMO if you're just scrolling this section trying to get an overview over directive handling (tool-specific directives are an important part of it).

fmease · 2025-06-02T17:39:33Z

src/rustdoc.md

-  where they're handled by the test runner of bootstrap and
-  the supplementary script `src/etc/htmldocck.py`.
-  [These tests have several extra directives available to them](./rustdoc-internals/rustdoc-test-suite.md).
-* Additionally, JavaScript type annotations are written using [TypeScript-flavored JSDoc]


This bullet point had nothing to do with tests (but with static analysis). I've moved it into Code Structure where the term JavaScript was already mentioned.

Ofc, that's still not great of a place, ideally we would have a separate section about internal linting / static analysis or something of similar nature.

the rationale was that both are run in CI, but I guess there is a fairly meaningful distinction between static analysis and unit/integration tests, with the line only becoming blurry again once you bring in proof assistants like kani.

fmease · 2025-06-02T17:45:30Z

src/rustdoc.md


-* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`.


This unordered list of tests suites was incredibly illegible. If you're trying to find out about a specific test suite, you basically needed to linearly scan through it and skip all that prose.

A table is way better. First I tried to turn it into a table, then I found out that we already have such a table over at tests/compiletest.md#rustdoc-test-suites. There's no need to duplicate this information, it's even counterproductive as everything gets easily outdated and it's hard to find a "single source of truth" because they don't match 100%.

So I've just removed this subsection (because this subpage shouldn't actually talk about testing, we have an entire directory full of documents for that¹) in favor of a cross-link. I've slightly fleshed out that table, so no information gets lost.

Footnotes

I really dislike how rustdoc content is split weirdly between rustdoc.md and rustdoc-internals.md as it's unclear where to put content. We need to reassess and restructure. Find a better flow. ↩

fmease · 2025-06-02T17:46:59Z

src/rustdoc-internals.md

@@ -270,35 +270,6 @@ in `test.rs` is the function `make_test`, which is where hand-written
 Some extra reading about `make_test` can be found
 [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).

-## Dotting i's And Crossing t's


Severely outdated (the rustdoc-js{,-std} test suites are nowadays fully documented in a separate document) and we also shouldn't talk about tests here, we have whole subdirectory dedicated for that.

lolbinarycat

Thanks for cleaning this up! I'm always in favor of deduplicating and peeling away unnecessary indirection, although leaving stubs with links to the main section is also good, for discovery, and I also appove of adding new stubs sections like the one you added about testsuite-specific directives.

lolbinarycat · 2025-06-02T19:51:45Z

src/rustdoc.md

 * The bits specific to using rustdoc as a test harness are in
  `doctest.rs`.
 * The Markdown renderer is loaded up in `html/markdown.rs`, including functions
  for extracting doctests from a given block of Markdown.
 * Frontend CSS and JavaScript are stored in `html/static/`.
+  * Re. JavaScript, type annotations are written using [TypeScript-flavored JSDoc]
+comments and an external `.d.ts` file.
+    This way, the code itself remains plain, valid JavaScript.


We should explain that this is so that tsc is not required as a build-time dependancy of rustdoc, not because we don't transform the source code at all (we are minifying it, so the jsdoc comments will not appear in the final documentation bundle unless minification is disabled, not sure how many of those details we should include)

isn't this duplicating some text that already exists somewhere? should we deduplicate that and make a subpage for how we use tsc? or is it small enough that the duplication isn't a problem?

lolbinarycat · 2025-06-02T19:54:26Z

src/rustdoc.md

-  where they're handled by the test runner of bootstrap and
-  the supplementary script `src/etc/htmldocck.py`.
-  [These tests have several extra directives available to them](./rustdoc-internals/rustdoc-test-suite.md).
-* Additionally, JavaScript type annotations are written using [TypeScript-flavored JSDoc]


the rationale was that both are run in CI, but I guess there is a fairly meaningful distinction between static analysis and unit/integration tests, with the line only becoming blurry again once you bring in proof assistants like kani.

fmease added S-waiting-on-author Status: this PR is waiting for additional action by the OP A-compiletest Area: src/tools/compiletest A-test-suite Area: rust-lang/rust test suites T-rustdoc Relevant to rustdoc team labels Mar 20, 2025

jieyouxu reviewed Mar 20, 2025

View reviewed changes

fmease mentioned this pull request Jun 2, 2025

Add link to correct documentation in htmldocck.py rust-lang/rust#141890

Open

fmease force-pushed the rustdoc-testing-addendum branch 2 times, most recently from c109003 to d276166 Compare June 2, 2025 17:22

[WIP] rustdoc testing: Further improve chapters and sections

d1b2c3f

fmease force-pushed the rustdoc-testing-addendum branch from d276166 to d1b2c3f Compare June 2, 2025 17:32

fmease commented Jun 2, 2025

View reviewed changes

lolbinarycat reviewed Jun 2, 2025

View reviewed changes

lolbinarycat mentioned this pull request Jun 2, 2025

add section about how rustdoc uses static analysis #2449

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] rustdoc testing: Further improve chapters and sections #2298

[WIP] rustdoc testing: Further improve chapters and sections #2298

Uh oh!

fmease commented Mar 20, 2025 •

edited

Loading

Uh oh!

jieyouxu Mar 20, 2025

Uh oh!

jieyouxu Mar 20, 2025

Uh oh!

jieyouxu Mar 20, 2025 •

edited

Loading

Uh oh!

jieyouxu Mar 20, 2025

Uh oh!

jieyouxu Mar 20, 2025

Uh oh!

fmease Jun 2, 2025 •

edited

Loading

Uh oh!

fmease Jun 2, 2025 •

edited

Loading

Uh oh!

lolbinarycat Jun 2, 2025

Uh oh!

fmease Jun 2, 2025 •

edited

Loading

Uh oh!

fmease Jun 2, 2025

Uh oh!

lolbinarycat left a comment

Uh oh!

lolbinarycat Jun 2, 2025

Uh oh!

lolbinarycat Jun 2, 2025

Uh oh!

Uh oh!

		@@ -34,33 +32,41 @@ In this case, the start of the next line should be `//`, with no `@`.

		For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root.

		<!-- FIXME(fmease): Mention that the regexes match case-sensitively and in single-line mode? -->

		htmldocck supports the `--bless` option to accept the current subtree
		Htmldocck supports the `--bless` option to accept the current subtree


		* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`.

[WIP] rustdoc testing: Further improve chapters and sections #2298

Are you sure you want to change the base?

[WIP] rustdoc testing: Further improve chapters and sections #2298

Uh oh!

Conversation

fmease commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jieyouxu Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Test infra setup

Uh oh!

fmease Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmease Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmease Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lolbinarycat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fmease commented Mar 20, 2025 •

edited

Loading

jieyouxu Mar 20, 2025 •

edited

Loading

fmease Jun 2, 2025 •

edited

Loading

fmease Jun 2, 2025 •

edited

Loading

fmease Jun 2, 2025 •

edited

Loading