Reverts #3291, which added our own robots.txt. I discovered that readthedocs will automatically generate an appropriate robots.txt based on which versions are activated and/or hidden. We can manage the visible versions from readthedocs.

The robots.txt file is at https://circuitpython.readthedocs.io/en/robots.txt.

See https://docs.readthedocs.io/en/stable/versions.html#states:

! Note
Active versions that are hidden will be listed as Disallow: /path/to/version/ in the default robots.txt file created by Read the Docs.

To manage visible versions, we use https://readthedocs.org/projects/circuitpython/versions/, and edit the individual versions. I just added 6.2.x, removed previous 6.x.x versions, and marked older versions as "hidden". Click on the Edit buttons to change the Active/Hidden state:

Example: