<div dir="ltr"><div>My apologies for the fxdev listspam. The alert_emails fields for Telemetry measurements are supposed to contain email addresses of individuals or small team lists, not firefox-dev. This probe
was written before we formalized our processes on this front... oh, and 26 others were as well.</div><div><br></div><div>I've filed bug 1470885 (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1470885">https://bugzilla.mozilla.org/show_bug.cgi?id=1470885</a>) to assign these probes better email addresses so that this shouldn't happen again.<br></div><div><br></div><div>While you're here...if you're interested in what's going on with this alert, you can see from the "
The value of FX_SANITIZE_TOTAL over time: <a href="https://telemetry.mozilla.org/new-pipeline/evo.html#!measure=FX_SANITIZE_TOTAL" rel="noreferrer" target="_blank">https://telemetry.mozilla.org/new-pipeline/evo.html#!measure=FX_SANITIZE_TOTAL</a>
" link that the number of submissions (pings) and samples (individual measurements per ping) underwent a significant downturn for Nightly builds between the 10th and 17th for some reason (#sfallhands2018). This resulted in an unrepresentative sample of sanitize timings to dominate the aggregate, which resulted in the distribution changing shape, which resulted in the alert being fired.</div><div><br></div><div>In short: We all stopped loading Nightly builds during All Hands, and that changed the shape of the graph. Go us! :mmayo must have really convinced us all to run Nightly last year, and we all had the good sense not to update Nightly over conference WiFi :)<br></div><div><br></div><div>The values and volumes are back to normal now, so we can call this an "Environmental Cause" and carry on with our work. No further action needed (outside of bug 1470885).<br></div><div><br></div><div>:chutten<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Sun, Jun 24, 2018 at 2:13 AM <<a href="mailto:telemetry-alerts@mozilla.com" target="_blank">telemetry-alerts@mozilla.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We have detected a change in the Telemetry probe FX_SANITIZE_TOTAL in Firefox Nightly builds from 2018-06-11.<br>
<br>
Alert details: <a href="http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/1134/alerts/?from=2018-06-11&to=2018-06-11" rel="noreferrer" target="_blank">http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/1134/alerts/?from=2018-06-11&to=2018-06-11</a><br>
Changes new to Nightly builds on 2018-06-11: <a href="https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8ab6afabc78cd909ff90ba74c1eab098985f83ef&tochange=62ab31ea0ec73e72a1ea9d44c9c4b003813b6724" rel="noreferrer" target="_blank">https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8ab6afabc78cd909ff90ba74c1eab098985f83ef&tochange=62ab31ea0ec73e72a1ea9d44c9c4b003813b6724</a><br>
The value of FX_SANITIZE_TOTAL over time: <a href="https://telemetry.mozilla.org/new-pipeline/evo.html#!measure=FX_SANITIZE_TOTAL" rel="noreferrer" target="_blank">https://telemetry.mozilla.org/new-pipeline/evo.html#!measure=FX_SANITIZE_TOTAL</a><br>
<br>
What to do about this:<br>
<br>
1. File a bug to track your investigation. You can just copy this email into the bug Description to get you started.<br>
2. Reply-All to this email to let the list know that you are investigating. Include the bug number so we can help out.<br>
3. Triage the alert. You can find instructions here: <a href="http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/1134/alerts/?from=2018-06-11&to=2018-06-11" rel="noreferrer" target="_blank">http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/1134/alerts/?from=2018-06-11&to=2018-06-11</a><br>
<br>
If you have any problems, please ask for help on the #telemetry IRC channel or on Slack in #fx-metrics. We'll give you a hand.<br>
<br>
What this is:<br>
We have a system called cerberus[1] that compares Telemetry collected on different Nightly builds and looks for sudden changed in value distributions using the Bhattacharyya Distance[2]. It found such a change in FX_SANITIZE_TOTAL on 2018-06-11 so it asked its buddy medusa[3] to send this email to the dev-telemetry-alerts mailing list and to all email addresses listed in the alert_emails field of FX_SANITIZE_TOTAL's definition.<br>
<br>
You can do this!<br>
<br>
Your Friendly, Neighbourhood Firefox Telemetry Team<br>
[1]: <a href="https://github.com/mozilla/cerberus" rel="noreferrer" target="_blank">https://github.com/mozilla/cerberus</a><br>
[2]: <a href="https://en.wikipedia.org/wiki/Bhattacharyya_distance" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/Bhattacharyya_distance</a><br>
[3]: <a href="https://github.com/mozilla/medusa" rel="noreferrer" target="_blank">https://github.com/mozilla/medusa</a><br>
_______________________________________________<br>
firefox-dev mailing list<br>
<a href="mailto:firefox-dev@mozilla.org" target="_blank">firefox-dev@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/firefox-dev" rel="noreferrer" target="_blank">https://mail.mozilla.org/listinfo/firefox-dev</a><br>
</blockquote></div>