<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
(If you're seeing this as a duplicate, note that the previous email,
sent to all users of Longitudinal, was sent to the wrong fx-data-dev
address.)<br>
<div class="moz-forward-container"><br>
Longitudinal users - we have found, and fixed, a bug in the
Longitudinal dataset.<br>
<p> Previous iterations of the longitudinal dataset were excluding
11% of users. We haven't done analysis on whether this was a
randomly selected group or if they all share some features or
usage patterns. Given the nature of the bug it is safe to assume
the latter. Details of the bug are available below.<br>
<br>
The next version of Longitudinal (2018-07-28) will have the fix,
and as such, your dashboards may look different! Additionally,
if you have any analysis/queries from the past 6 months that
relied on Longitudinal, we *strongly* recommend rerunning on the
new dataset (when it is available) - the results may have
changed. We will also be rebuilding historical versions of
Longitudinal for the past six months. <b>Once the data is
available I will reply-all to this email.</b><br>
<br>
Moving forward, we will see only around .0008% of clients
ignored (52 in the most recent tests); these are due to
malformed pings and are safe to ignore.<br>
<br>
We do expect to EOL the Longitudinal dataset within the coming
two quarters. If you rely on it for ongoing analysis, we will be
in contact with you about moving to a different datasource.</p>
<p>If you have <i>any</i> concerns, please contact us in
#fx-metrics on Slack or #telemetry in IRC.</p>
<p><br>
--Boring Details Below--<br>
<br>
The issue was that we tried to hash some Avro types by creating
a Set structure out of them. However, when looking up the
HashCode of the Avro type, 11% of clients had at least one ping
with a BigInteger or BigDecimal Avro type (after conversion).
That ended up hitting an Avro bug: <a
class="moz-txt-link-freetext"
href="https://issues.apache.org/jira/browse/AVRO-1146"
moz-do-not-send="true">https://issues.apache.org/jira/browse/AVRO-1146</a><br>
<br>
Fix:
<a class="moz-txt-link-freetext"
href="https://github.com/mozilla/telemetry-batch-view/commit/8a8525b5a980908907b63f38017f82fa4ea10ab8"
moz-do-not-send="true">https://github.com/mozilla/telemetry-batch-view/commit/8a8525b5a980908907b63f38017f82fa4ea10ab8</a><br>
<br>
Verification of the bug:<br>
Longitudinal Client Count: <a class="moz-txt-link-freetext"
href="https://sql.telemetry.mozilla.org/queries/57350/source"
moz-do-not-send="true">https://sql.telemetry.mozilla.org/queries/57350/source</a><br>
Main Summary Client Count, same period: <a
class="moz-txt-link-freetext"
href="https://sql.telemetry.mozilla.org/queries/57349/source"
moz-do-not-send="true">https://sql.telemetry.mozilla.org/queries/57349/source</a></p>
<p><br>
-Frank Bertsch (:frank)<br>
</p>
</div>
</body>
</html>