Mirroring AMO history (was Re: FF addons)

The Wanderer wanderer at fastmail.fm
Tue Aug 8 15:42:55 UTC 2017

On 2017-08-07 at 17:22, Jorge Villalobos wrote:

> On 8/7/17 5:51 AM, The Wanderer wrote:
>> On 2017-08-04 at 13:37, Jorge Villalobos wrote:

>>> We have no plans for this. The Internet Archive keeps good 
>>> historical records of AMO, but I don't know how they fare with 
>>> installer files.
>> How much space would be required for a full mirror of AMO (if
>> necessary limited to data-only, i.e. without any of the code which
>> runs the site), and how would one go about requesting a dump /
>> export of the current archive (including historical versions) in
>> order to create such a mirror?
> I think you can use the API[1] to enumerate all listed add-ons, and
> the download URL for the latest version is easily predictable using
> the ID.

I'm actually interested in the full history of all listed versions, as
far back as AMO currently hosts - but indeed, it looks as if it's
possible to get that from the API as well, albeit with added complexity.

(Unlisted and otherwise private versions would be nice to archive as
well, in case a previous author comes back and is dismayed to find that
something in that category is no longer accessible, but I can readily
understand why I wouldn't get access to download those.)

I probably wouldn't have been able to readily put together something to
interact with that API usefully (almost all of my coding experience is
either C or, more recently, JS), but apparently someone else has already
done most of the work; [2], which someone pointed me to in private mail,
doesn't quite do what I need as-is (for one thing, the download stage
errors out immediately) but can be hacked up to get the job done.

Any ballpark figures for how much space this would require? I'm
reasonably sure I have enough and to spare, but I'd rather not start the
mass download and discover three days later that I've run out of disk
space without even finishing the job.

> [1] http://addons-server.readthedocs.io/en/latest/topics/api/addons.html

[2] https://github.com/andymckay/addons-server-mirror

   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.mozilla.org/pipermail/dev-addons/attachments/20170808/58c08b2b/attachment.sig>

More information about the Dev-addons mailing list