PSA: Recent Telemetry client issues

[Georg Fritzsche]

Hi,

We ran into three issues with Firefox Desktop Telemetry in February that we
want to make people aware of.

Sync ping not sent on Firefox shutdown

The sync team noticed a lack of their sync pings around Firefox shutdown in
Firefox 58+. This turned out to be a race in Telemetry shutdown, which lead
to the sync pings not being saved in time.

Other critical pings like the main ping were not affected. For sync, this
was limited to the pings sent at Firefox shutdown.

This is fixed on Firefox 59+ in bug 1437120
<https://bugzilla.mozilla.org/show_bug.cgi?id=1437120>, validation is
happening in bug 1439564
<https://bugzilla.mozilla.org/show_bug.cgi?id=1439564>.

Recording large histogram values overflows

As Chris investigated a curious issue
<https://chuttenblog.wordpress.com/2018/02/23/perplexing-graphs-the-case-of-the-0kb-virtual-memory-allocations/>,
it became clear that we had a silent historic problem: accumulating values
outside the supported histogram range over-/underflows and will be recorded
into the wrong bucket.

We fixed this in bug 1438335
<https://bugzilla.mozilla.org/show_bug.cgi?id=1438335> by 1) clamping the
values <https://hg.mozilla.org/integration/autoland/rev/a06841d0b44a> to
the min or max bucket, 2) logging an error and 3) reporting this
<https://hg.mozilla.org/integration/autoland/rev/a06841d0b44a> in Telemetry.

This fix landed on Firefox 60, we're looking at uplift to Firefox 59.

We're still waiting for some more data to come in to draw final
conclusions, but early data <https://mzl.la/2CmmpmK> suggests that no
critical histograms were affected.

Missing parts of prerelease data from Beta channel

We noticed that since Firefox 58 we are not collecting prerelease data from
"late beta" builds (release candidate builds), which are build with a
"release" configuration, which is effectively the last third of the Firefox
beta cycle.

This is a side-effect of the recent data preference changes
<https://bugzilla.mozilla.org/show_bug.cgi?id=1406390>.

We are landing a fix in bug 1406390
<https://bugzilla.mozilla.org/show_bug.cgi?id=1406390> in Firefox 60.

What's next?

We are still following up on these issues currently and will do a
retrospective on them soon after.

In the meantime, please reach out to us (:gfritzsche, :chutten and the
larger team) with any questions.

Cheers,

Georg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/fx-data-dev/attachments/20180227/4cc72bf4/attachment.html>