<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Awesome, Gene. I'm counting on it.</div><div><br></div><div>-chris</div><div><br></div><br><div><div>On Nov 5, 2013, at 1:35 PM, Gene Wood <<a href="mailto:gene@mozilla.com">gene@mozilla.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">I'll also be prototyping cross region master slave MySQL (like we have in Persona) in the background that we can swap in when we want to go multi-region.<div>-Gene</div></div><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Tue, Nov 5, 2013 at 1:23 PM, Benson Wong <span dir="ltr"><<a href="mailto:bwong@mozilla.com" target="_blank">bwong@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Lloyd,<br>
<br>
Some details on my rational on the ops side. Yes, it is to make things go faster. It's also to make things simpler while we get a grasp on what we really need to run the service.<br>
<br>
Some of my current ideas on HA for the service:<br>
<br>
On API HA:<br>
<br>
- RDS w/ hot standby (synchronous replication).<br>
- RDS snapshots shipped to another region. New feature, automated via API. Frequency TBD<br>
- Start in us-west-2, which AFAIK, has been a very stable region.<br>
<br>
The scariest scenario is when an entire region goes down and doesn't come back for many hours. In these cases, our recovery procedure will be:<br>
<br>
1. Decide: swap regions, or wait?<br>
2. Do it.<br>
3. Decide: swap back, or leave it?<br>
<br>
On Content/Scrypt servers:<br>
<br>
- these are stateless AFAIK. So we can spin these up anywhere and re-point DNS quickly.<br>
- the goal here will be making it dead simple to spin up, and redirect traffic.<br>
<br>
Hope that answers your questions (and makes you less nervous).<br>
<br>
Ben.<br>
<div class="HOEnZb"><div class="h5"><br>
----- Original Message -----<br>
From: "Lloyd Hilaiel" <<a href="mailto:lhilaiel@mozilla.com">lhilaiel@mozilla.com</a>><br>
To: "Christopher Karlof" <<a href="mailto:ckarlof@mozilla.com">ckarlof@mozilla.com</a>><br>
Cc: "Benson Wong" <<a href="mailto:bwong@mozilla.com">bwong@mozilla.com</a>>, "Ryan Kelly" <<a href="mailto:rfkelly@mozilla.com">rfkelly@mozilla.com</a>>, "Mozilla Services Operations" <<a href="mailto:services-ops@mozilla.com">services-ops@mozilla.com</a>>, <a href="mailto:dev-fxacct@mozilla.org">dev-fxacct@mozilla.org</a>, "Gene Wood" <<a href="mailto:gene@mozilla.com">gene@mozilla.com</a>><br>
Sent: Tuesday, November 5, 2013 3:58:08 AM<br>
Subject: Re: Firefox Accounts DevOps next steps for November<br>
<br>
<br>
Not going multi-region from day one makes me nervous. Technology selections which make it harder make me even more nervous. Can we hit HA requirements without it?<br>
<br>
Is the rationale here simply to accelerate timelines?<br>
<br>
lloyd<br>
</div></div></blockquote></div><br></div>
</blockquote></div><br></body></html>