[rust-dev] robots.txt prevents Archive.org from storing old documentation

Gioele Barabucci gioele at svario.it
Thu Jul 10 00:46:18 PDT 2014


Hi,

the current robots.txt on docs.rust-lang.org prevents Archive.org from 
storing copies of the old documentation. I think having the old 
documentation archived would be a good thing. BTW, all the documentation 
before 0.10 seems gone and this is a shame.

Could you please allow the Archive.org bot to index the site?

For the records:

     $ curl http://doc.rust-lang.org/robots.txt
     User-agent: *
     Disallow: /0.3/
     Disallow: /0.4/
     Disallow: /0.5/
     Disallow: /0.6/
     Disallow: /0.7/
     Disallow: /0.8/
     Disallow: /0.9/
     Disallow: /0.10/

-- 
Gioele Barabucci <gioele at svario.it>


More information about the Rust-dev mailing list