[rust-dev] robots.txt prevents Archive.org from storing old documentation

Chris Morgan me at chrismorgan.info
Mon Jul 14 18:56:44 PDT 2014


On Tue, Jul 15, 2014 at 4:16 AM, Brian Anderson <banderson at mozilla.com> wrote:
> Can somebody file an issue described exactly what we should do and cc me?

Nothing. Absolutely nothing.

robots.txt rules do not apply to historical data; if archive.org has
archived something, the introduction of a new Disallow rule will not
remove the contents of a previous scan.

It therefore has three months in which to make a scan of a release
before that release is marked obsolete with the introduction of a
Disallow directive.

This is right and proper. Special casing a specific user agent is not
the right thing to do. The contents won’t be changing after the
release, anyway, so allowing archive.org to continue scanning it is a
complete waste of effort.


More information about the Rust-dev mailing list