<div dir="ltr"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" id="gmail-docs-internal-guid-256a93d9-d8a8-bc68-daee-ac52651e3d78"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"></span></font><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Hi,</span></font><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We ran into three issues with Firefox Desktop Telemetry in February that we want to make people aware of.</span></font></p><font size="2"><br></font><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Sync ping not sent on Firefox shutdown</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">The sync team noticed a lack of their sync pings around Firefox shutdown in Firefox 58+. This turned out to be a race in Telemetry shutdown, which lead to the sync pings not being saved in time.</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Other critical pings like the main ping were not affected. For sync, this was limited to the pings sent at Firefox shutdown.</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">This is fixed on Firefox 59+ in </span><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1437120" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">bug 1437120</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">, validation is happening in </span><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1439564" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">bug 1439564</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">.</span></font></p><font size="2"><br></font><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Recording large histogram values overflows</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">As Chris investigated a </span><a href="https://chuttenblog.wordpress.com/2018/02/23/perplexing-graphs-the-case-of-the-0kb-virtual-memory-allocations/" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">curious issue</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">, it became clear that we had a silent historic problem: accumulating values outside the supported histogram range over-/underflows and will be recorded into the wrong bucket.</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We fixed this in </span><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1438335" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">bug 1438335</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> by 1) </span><a href="https://hg.mozilla.org/integration/autoland/rev/a06841d0b44a" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">clamping the values</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> to the min or max bucket, 2) logging an error and 3) </span><a href="https://hg.mozilla.org/integration/autoland/rev/a06841d0b44a" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">reporting this</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> in Telemetry.</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">This fix landed on Firefox 60, we're looking at uplift to Firefox 59.</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We're still waiting for some more data to come in to draw final conclusions, but </span><a href="https://mzl.la/2CmmpmK" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">early data</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> suggests that no critical histograms were affected.</span></font></p><font size="2"><br></font><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Missing parts of prerelease data from Beta channel</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We noticed that since Firefox 58 we are not collecting prerelease data from "late beta" builds (release candidate builds), which are build with a "release" configuration, which is effectively the last third of the Firefox beta cycle.</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">This is a side-effect of the recent </span><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1406390" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">data preference changes</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">.</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We are landing a fix in </span><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1406390" style="text-decoration:none"><span style="font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">bug 1406390</span></a><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> in Firefox 60.</span></font></p><font size="2"><br></font><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">What's next?</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We are still following up on these issues currently and will do a retrospective on them soon after.</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">In the meantime, please reach out to us (:gfritzsche, :chutten and the larger team) with any questions.</span></font></p><font size="2"><br></font><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Cheers,</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Georg</span></font></p><br></div>