[rust-dev] robots.txt prevents Archive.org from storing old documentation

Brian Anderson banderson at mozilla.com
Mon Jul 14 11:16:43 PDT 2014


Can somebody file an issue described exactly what we should do and cc me?

On 07/14/2014 01:13 AM, richo wrote:
> On 10/07/14 17:15 +0000, Ivan Kozik wrote:
>> On Thu, Jul 10, 2014 at 3:49 PM, Jonas Wielicki
>> <j.wielicki at sotecware.net> wrote:
>>> While this is a good thing /all/ software projects should be doing imo,
>>> one could still explicitly allow Archive.org by prepending:
>>>
>>>     User-agent: ia_archiver
>>>     Disallow:
>>>
>>> ?
>>
>> It looks like documentation for the old versions e.g.
>> http://doc.rust-lang.org/0.9/ is a 404 anyway.
>>
>
> Which is kinda the point of letting archive.org keep a copy for 
> posterity.
>
> Not that I have a bunch of sway, but I'm +1 on not letting old docs be
> searchable, and also +1 on making an exception for archive.org
> _______________________________________________
> Rust-dev mailing list
> Rust-dev at mozilla.org
> https://mail.mozilla.org/listinfo/rust-dev



More information about the Rust-dev mailing list