<div dir="ltr">I'll mirror those in the bugs, but it is evident that the t1.micros we threw in last week were not up to handling thimble.<div><img src="cid:ii_14421e162908cbe0" alt="Inline image 1" width="420" height="233"><br>
</div><div><br></div><div>So then the question becomes: What is stressing the db? We have a few clues to go on, and ways to get more clues.</div><div><br></div><div>1) The LegacyThimble database likely needs some indexing for use on the /p/:oldid path. The smaller instance size exacerbated the issue, but this looks like a potential root cause of stressing the db. </div>
<div><img src="cid:ii_14421e3e53b54d89" alt="Inline image 2" width="420" height="304"><br></div><div>2) Slow query log is currently off, and we can enable it. However, this will take a minute or so of downtime, so I am holding off during the day.</div>
<div><br></div><div>3) I have a script running on our monitoring box that is checking the processlist every 20 seconds to try to grab the offending query.</div><div><br></div><div><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Feb 11, 2014 at 10:50 AM, Jon Buckley <span dir="ltr"><<a href="mailto:jon@mozillafoundation.org" target="_blank">jon@mozillafoundation.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><div>JP is putting his findings in <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=970833" target="_blank">https://bugzilla.mozilla.org/show_bug.cgi?id=970833</a></div><div><div class="h5">
<br><div><div>On Feb 11, 2014, at 11:03 AM, David Ascher <<a href="mailto:da@mozilla.com" target="_blank">da@mozilla.com</a>> wrote:</div><br><blockquote type="cite"><div style="word-wrap:break-word">do we know why the db was pegged?<div>
<br></div><div>—da</div><div><br><div><div>On Feb 11, 2014, at 7:11 AM, David Humphrey <<a href="mailto:david.humphrey@senecacollege.ca" target="_blank">david.humphrey@senecacollege.ca</a>> wrote:</div><br><blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">
<div>Great work, JP, thanks.<br>
<br>
On 14-02-11 8:28 AM, JP Schneider wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I have now fully recovered thimble. I did so by
resizing the existing thimble database, so we have had no data
loss.
<div><br>
</div>
<div>In total, we were down from <span style="font-family:arial,sans-serif;font-size:13.333333015441895px">12:17:22(UTC)
- 13:26:00(UTC). The app itself was available, but no user
data was able to make it to our database.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13.333333015441895px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13.333333015441895px"><br>
</span></div>
</div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On Tue, Feb 11, 2014 at 7:20 AM, JP
Schneider <span dir="ltr"><<a href="mailto:jp@mozillafoundation.org" target="_blank">jp@mozillafoundation.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Now that I am the process of recovering this,
some additional details.
<div><br>
</div>
<div>It looks like our db was a bit stressed:</div>
<div><span><ATT00001.jpeg></span><br>
</div>
<div><br>
</div>
<div>After forcing a failover, I was able to connect once
again.</div>
<div>I am currently resizing this instance back to an
m1.small.</div>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On Tue, Feb 11, 2014 at 7:02
AM, JP Schneider <span dir="ltr"><<a href="mailto:jp@mozillafoundation.org" target="_blank">jp@mozillafoundation.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Good morning,
<div><br>
</div>
<div>There seems to have been some corruption in
our Thimble production database. I am looking
into this now.</div>
<div><br>
</div>
<div>The symptom to the end user seems to be
when an item is published, no URL is given for
your new make.</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
_______________________________________________<br>Webmaker-dev mailing list<br><a href="mailto:Webmaker-dev@mozilla.org" target="_blank">Webmaker-dev@mozilla.org</a><br><a href="https://mail.mozilla.org/listinfo/webmaker-dev" target="_blank">https://mail.mozilla.org/listinfo/webmaker-dev</a><br>
</blockquote></div><br></div></div>_______________________________________________<br>Webmaker-dev mailing list<br><a href="mailto:Webmaker-dev@mozilla.org" target="_blank">Webmaker-dev@mozilla.org</a><br><a href="https://mail.mozilla.org/listinfo/webmaker-dev" target="_blank">https://mail.mozilla.org/listinfo/webmaker-dev</a><br>
</blockquote></div><br></div></div></div><br>_______________________________________________<br>
Webmaker-dev mailing list<br>
<a href="mailto:Webmaker-dev@mozilla.org">Webmaker-dev@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/webmaker-dev" target="_blank">https://mail.mozilla.org/listinfo/webmaker-dev</a><br>
<br></blockquote></div><br></div>