[rust-dev] robots.txt prevents Archive.org from storing old documentation

Daniel Micay danielmicay at gmail.com
Thu Jul 10 07:56:33 PDT 2014


On 10/07/14 03:46 AM, Gioele Barabucci wrote:
> Hi,
> 
> the current robots.txt on docs.rust-lang.org prevents Archive.org from
> storing copies of the old documentation. I think having the old
> documentation archived would be a good thing. BTW, all the documentation
> before 0.10 seems gone and this is a shame.
> 
> Could you please allow the Archive.org bot to index the site?
> 
> For the records:
> 
>     $ curl http://doc.rust-lang.org/robots.txt
>     User-agent: *
>     Disallow: /0.3/
>     Disallow: /0.4/
>     Disallow: /0.5/
>     Disallow: /0.6/
>     Disallow: /0.7/
>     Disallow: /0.8/
>     Disallow: /0.9/
>     Disallow: /0.10/

The old documentation is all available from the Git repository. The
robots.txt rule is there to reverse the trend of searches being filled
with out of date documentation.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.mozilla.org/pipermail/rust-dev/attachments/20140710/089e49dc/attachment.sig>


More information about the Rust-dev mailing list