[rust-dev] robots.txt prevents Archive.org from storing old documentation
me at chrismorgan.info
Mon Jul 14 18:56:44 PDT 2014
On Tue, Jul 15, 2014 at 4:16 AM, Brian Anderson <banderson at mozilla.com> wrote:
> Can somebody file an issue described exactly what we should do and cc me?
Nothing. Absolutely nothing.
robots.txt rules do not apply to historical data; if archive.org has
archived something, the introduction of a new Disallow rule will not
remove the contents of a previous scan.
It therefore has three months in which to make a scan of a release
before that release is marked obsolete with the introduction of a
This is right and proper. Special casing a specific user agent is not
the right thing to do. The contents won’t be changing after the
release, anyway, so allowing archive.org to continue scanning it is a
complete waste of effort.
More information about the Rust-dev