Firefox Accounts DevOps next steps for November

Chris Karlof ckarlof at mozilla.com
Tue Nov 5 15:07:56 PST 2013


On Nov 5, 2013, at 1:23 PM, Benson Wong <bwong at mozilla.com> wrote:

> Hi Lloyd, 
> 
> Some details on my rational on the ops side. Yes, it is to make things go faster. It's also to make things simpler while we get a grasp on what we really need to run the service. 
> 
> Some of my current ideas on HA for the service: 
> 
> On API HA:
> 
> - RDS w/ hot standby (synchronous replication).
> - RDS snapshots shipped to another region. New feature, automated via API. Frequency TBD
> - Start in us-west-2, which AFAIK, has been a very stable region.
> 
> The scariest scenario is when an entire region goes down and doesn't come back for many hours. In these cases, our recovery procedure will be: 
> 
> 1. Decide: swap regions, or wait?
> 2. Do it. 
> 3. Decide: swap back, or leave it? 
> 
> On Content/Scrypt servers: 
> 
> - these are stateless AFAIK. So we can spin these up anywhere and re-point DNS quickly.
> - the goal here will be making it dead simple to spin up, and redirect traffic.
> 
> Hope that answers your questions (and makes you less nervous). 
> 

Nice summary Ben. 

Lloyd: working and simple first, then pimp it out.

I'm confident we can easily pivot on the basic architecture Gene and Benson are putting together to a multi-region MySQL setup or away from RDS if need be. Gene or Benson, if you disagree, speak up.

-chris



> Ben.
> 
> ----- Original Message -----
> From: "Lloyd Hilaiel" <lhilaiel at mozilla.com>
> To: "Christopher Karlof" <ckarlof at mozilla.com>
> Cc: "Benson Wong" <bwong at mozilla.com>, "Ryan Kelly" <rfkelly at mozilla.com>, "Mozilla Services Operations" <services-ops at mozilla.com>, dev-fxacct at mozilla.org, "Gene Wood" <gene at mozilla.com>
> Sent: Tuesday, November 5, 2013 3:58:08 AM
> Subject: Re: Firefox Accounts DevOps next steps for November
> 
> 
> Not going multi-region from day one makes me nervous.  Technology selections which make it harder make me even more nervous.  Can we hit HA requirements without it?  
> 
> Is the rationale here simply to accelerate timelines?
> 
> lloyd




More information about the Dev-fxacct mailing list