<div dir="ltr"><div><div><div><div><div>This message is going to cover a lot of stuff, so you might want to grab a cup of coffee now :)<br><br></div>So first, even though it may seem like I'm strongly arguing against the Telemetry/FHR v4 approach, I'm really trying to figure out how all the different Telemetry use-cases are going to be affected by the new ping format. And I'm also trying to figure out a reasonable plan of attack.<br><br></div><div></div>I think the biggest "unknown" in the current discussions is the design of the new Telemetry backend and whether it will be able to stitch together user sessions and "user-days" efficiently. <br>I think the backend will require significant changes. The pings will have to be grouped and stored differently. We'll have to write a LOT of new backend code for "stitching" pings, feeding the dashboards, supporting custom analyses, etc. We'll need to create a prototype of the new Telemetry/FHR backend and test it against realistic data volumes before landing major client-side changes. Migrating existing Telemetry (and FHR?) pings to the new backend will be
impossible, so we'll have to run two systems for a while. We'll also
need to continue supporting the old Telemetry (and FHR?) formats for
Fennec & B2G.<br><br></div>The timing for this change is bad because Perf team will need Telemetry in Q1 to evaluate E10S & Flash performance, not to mention other teams relying on Telemetry and FHR data for their projects. So I think it's a mistake to risk significant FHR & Telemetry downtime while we re-write the entire backend.<br><br></div>I think we should try to land the client-side changes gradually. <br><br></div>A rough sketch:<br><div><ol><li>Modify the server backend to accept Telemetry pings in the new JSON format (clientID, sessionID, sysinfo, env, etc). Bump version number.</li><ul><li>Also update the backend to parse E10S data from the Telemetry payload<br></li><li>Update dashboard code<br></li></ul><li>Modify client to send Telemetry pings in the new JSON format. Rip out the obsolete "idle-daily" Telemetry ping.<br></li><ul><li>Continue submitting Telemetry saved-session ping </li><li>Continue to upload FHR data separately</li></ul><li>Modify the Telemetry backend code to ignore all mid-session Telemetry pings.<br></li><li>Modify Telemetry client-side to create new pings on 24-hour boundaries. Do not reset any measurements in the middle of a session. Indicate the last ping of a session explicitly. Bump version number.</li><ul><li>This is essentially my earlier proposal, but as a (hopefully) temporary measure.</li><li>E10S child Telemetry code will have to be modified to create subsessions at the same time as parent</li></ul><li>Adapt Telemetry Experiments backend as needed<br></li><li>Modify Telemetry code to create new pings on environment change. Update Experiments code as needed<br></li><li>Modify backend to parse and store the new unified ping format. Stress test etc.</li><ul><li>All FHR backend functionality should be complete before the next step</li><li>The FHR user-day stitching code should be finished and tested with realistic loads</li></ul><li>Switch FHR to use the Telemetry upload mechanism, and switch over FHR & Telemetry to the new unified ping format. Reset FHR measurements on new pings. Bump version number</li><ul><li>At this point, the people working on the FHR backend can stabilize and improve the backend</li><li>After we have experience with handling the new ping semantics with FHR, we can move on to converting Telemetry in the next step<br></li></ul><li>Assuming success of step 7, write and test session stitching, extend FHR user-day stitching to Telemetry data, integrate new Telemetry subsession format with map-reduce and Spark analysis, dashboards, regression detector, etc.</li><ul><li>Lots of custom stitching rules, e.g. Background-Hang-Reporter data<br></li></ul><li>Change Telemetry to reset-on-new-ping semantics. Adapt existing probes. Bump version number.</li><ul><li>about:telemetry will be a pain to convert. It will have to do client-side stitching using saved local pings, or display the subsession separately. Both are bad<br></li></ul><ul><li>fx-team's UITelemetry reporting will require some special
attention. It's very much session-based and there are no clear rules for
stitching it together from subsessions</li><li>Identify and fix one-per-session histograms<br></li><li>Document and publicize the new ping model for Telemetry probe authors<br></li></ul></ol><p>I think this gradual approach would allows us to focus on converting one backend at a time and to use the experience gained with FHR to convert Telemetry.</p><p>What do you think?</p><p>Vladan<br></p><p></p></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jan 25, 2015 at 11:58 AM, Benjamin Smedberg <span dir="ltr"><<a href="mailto:benjamin@smedbergs.us" target="_blank">benjamin@smedbergs.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class="">
<br>
<br>
Georg wrote:<br>
<blockquote type="cite">My assumptions was that we will not reset. </blockquote>
<br></span>
The proposal as written is that we will reset all the histograms for
each subsession. Otherwise, realtime dashboard which process
incoming pings will multipe-count various metrics, and we definitely
want to avoid this.<span class=""><br>
<br>
<div>On 1/24/2015 1:35 AM, Vladan Djeric
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div>
<ul>
<li>It will be hard to do per-session analyses</li>
</ul>
</div>
</div>
</blockquote>
<br></span>
I have several responses here:<br>
<br>
1) It will be a bit harder than currently, but I don't think that it
will be extremely hard. There will be an efficient API to fetch all
the pings associated with a user, which should make it relatively
straightforward to stitch together an entire session from its
pieces. This is a functional requirement for the more qualitative
analyses, which will have to stitch together an entire user history
and not just individual sessions. Doing an individual session should
be fairly easy.<br>
<br>
2) I treat the session orientation of telemetry as an unfortunate
limitation, not a desirable property, for almost all of the use
cases that I've seen. I'd like us to try and move away from
reporting metrics based on sessions. Can you describe in more detail
the use cases where analyzing data by session is preferable to
analyzing by some constant denominator? We should be willing to use
both clock time and activeTicks as denominators, and these
denominators can both be calculated looking at individual subsession
pings.<br>
<br>
3) For the case of the current telemetry dashboard, I'd like to
understand why simply replacing the current whole-session analysis
with the new subsessions would produce statistically worse results
than the current session-based analysis.<span class=""><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<ul>
<li>Many of the 1000+ Telemetry measurements are inherently
"per-session" and can't meaningfully be split into session
fragments:<br>
</li>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type" target="_blank">Flag
histograms</a> track feature usage per-session.<br>
</li>
<ul>
<li>They are automatically initialized to a value of
"false" at the beginning of a session, and can only be
set to "true" once.<br>
</li>
<li><span style="color:rgb(204,0,0)">If we reset
Telemetry measurements every time we create a new
ping, we'll be reporting nonsense: pings from the
same session will contradict each other on whether a
feature was ever used during the session</span><br>
</li>
<ul>
<li>This would feed bad data to both the dashboards
and any custom analyses<br>
</li>
</ul>
</ul>
</ul>
</ul>
</div>
</div>
</blockquote></span>
*If* you really care about this per-session, why can't you just take
"true" from any of the subsessions as an indication that it's true
for the entire session?<br>
<br>
And if we just report by subsession, how is this much different from
the skew that we already have between users who have lots of short
sessions and users that keep their browser open for days or weeks?<br>
<br>
Maybe this just indicates that we're mis-using histograms for
non-aggregate measurements, and we should just have a separate list
of flag metrics which are treated differently.<span class=""><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<ul>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type" target="_blank">Count
histograms</a> are also per-session measurements. You
can't aggregate a count-histogram value from the middle
of a session together with final values from other
sessions<br>
</li>
</ul>
</ul>
</div>
</div>
</blockquote>
<br></span>
Won't summing across the subsessions get you the total count for the
session?<span class=""><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<ul>
<li>For custom analyses, we sometimes want to correlate
measurements from the beginning of a session with
measurements from the end of a session (which could have
lasted several days), e.g. histograms related to startup
performance vs later performance</li>
<ul>
<li>We would need that messy server-side session
reconstruction process to get at per-session data.</li>
<li>More generally, a ping generated as a result of local
time & environment changes is
not inherently meaningful to us, unlike a full user
session</li>
</ul>
</ul>
</div>
</div>
</blockquote>
<br></span>
I don't understand this case. Assuming session stitching works,
which is a general requirement for all sorts of analyses, this
should work no worse than currently, and you potentially have
finer-grain data on the subsequent days if that's useful.<span class=""><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<ul>
<li>Resetting Telemetry and FHR data when a
TelemetryExperiment begins removes valuable context from
the experiment ping. It's possible to reconstruct it, but
that's yet another server-side job to run</li>
</ul>
</div>
</div>
</blockquote></span>
I don't understand this. Is this also assuming that stitching is
expensive?<span class=""><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<ul>
<li>There's overhead from sending a new ping for each
mid-session environment change</li>
<ul>
<li>There's also a small privacy issue with creating
ordered, fine-grained reports of user actions, e.g. when
a user goes through their add-ons list and disables 5
addons, we report each user action</li>
<li>Either coalesce successive environment-change pings,
or carefully vet which mid-session environment changes
generate a new ping</li>
</ul>
</ul>
</div>
</div>
</blockquote>
<br></span>
I think it's worth considering whether there's a window of time
where multiple changes get coalesced. But I'm not particularly
worried about the privacy problem, since we do in fact want to
record when users disable addons.<span class=""><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>I'd like to propose that we implement the following
modifications to the FHR/Telemetry v4 document:<br>
<ol>
<li>Do not reset <u>Telemetry</u> measurements when a
session crosses the 24-hour boundary</li>
<ul>
<li>Continue to "reset" Telemetry measurements when we
start a new session</li>
<li>There's no need to reset Telemetry on most environment
changes (e.g. amount of memory installed) since those
can't happen without a Firefox restart anyway.</li>
</ul>
<li>Record mid-session environment changes (add-ons and
TelemetryExperiments) in a special section in the ping.</li>
<ul>
<li>For each such environment change, document the change
in the section and also attach a snapshot of the
Telemetry & FHR data at the time of the change</li>
<li>After the snapshot is saved, reset Telemetry and FHR
measurements for the current session. In other words,
snapshot & then build up a diff<br>
</li>
<li>For each additional environment change during the same
session, just repeat and append to the new section<br>
</li>
<li>Telemetry backend scripts (dashboard, regression
detector etc) can just ignore experiment/add-on change
pings</li>
</ul>
</ol>
This model has some nice properties:</div>
<div>
<ul>
<li>The <b>final ping</b> of a session is equivalent to a
Telemetry saved-session ping</li>
<ul>
<li>Per-session analyses are as easy to do as before</li>
</ul>
<ul>
<li>No need to run any session reconstruction jobs!</li>
</ul>
<li>Every main ping submitted is meaningful without needing
any reconstruction steps. All pings will contain the
current FHR state + all the Telemetry measurements from
the current session<br>
</li>
<li>Most pings will only have one environment change, so the
relevant measurements that happened after the change are
all going to be in the regular Telemetry/FHR section</li>
<li>However, when deeper analysis is required, Experiment
pings will also have information about what was happening
BEFORE the experiment began</li>
<li>Analyzing pings with multiple environment changes won't
be much harder</li>
</ul>
</div>
</div>
</blockquote>
<br></span>
I feel like this proposal is optimizing for the wrong things.<br>
<br>
You are making a distinction between "Telemetry" measurements and
other measurements in a way which I am specifically trying to avoid.
The goal is to use the common histogram system for everything. At
least some of those measurements must be distinguished by
subsession. I explicitly want to get rid of the current situation
where "telemetry metrics" are treated one way, and "FHR metrics" are
treated in some entirely separate manner. We want to be able to use
the standard histograms/keyed histograms for almost everything.<br>
<br>
For the simple things like the telemetry dashboard, I believe that
doing all analysis by subsession is good enough (no worse than the
current situation). For more complex queries , both stitching
together an entire session and stitching together the history
per-user will not only be possible but should be fairly efficient.<br>
<br>
--BDS<br>
<br>
</div>
</blockquote></div><br></div></div>