[rust-dev] robots.txt prevents Archive.org from storing old documentation

richo richo at psych0tik.net
Mon Jul 14 01:13:46 PDT 2014

On 10/07/14 17:15 +0000, Ivan Kozik wrote:
>On Thu, Jul 10, 2014 at 3:49 PM, Jonas Wielicki
><j.wielicki at sotecware.net> wrote:
>> While this is a good thing /all/ software projects should be doing imo,
>> one could still explicitly allow Archive.org by prepending:
>>     User-agent: ia_archiver
>>     Disallow:
>> ?
>It looks like documentation for the old versions e.g.
>http://doc.rust-lang.org/0.9/ is a 404 anyway.

Which is kinda the point of letting archive.org keep a copy for posterity.

Not that I have a bunch of sway, but I'm +1 on not letting old docs be
searchable, and also +1 on making an exception for archive.org

More information about the Rust-dev mailing list