<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
Georg wrote:<br>
<blockquote type="cite">My assumptions was that we will not reset. </blockquote>
<br>
The proposal as written is that we will reset all the histograms for
each subsession. Otherwise, realtime dashboard which process
incoming pings will multipe-count various metrics, and we definitely
want to avoid this.<br>
<br>
<div class="moz-cite-prefix">On 1/24/2015 1:35 AM, Vladan Djeric
wrote:<br>
</div>
<blockquote
cite="mid:CAFsdOVSuZ29a+ePKJbQgrOazkx9YU2La9cvvEALA96Lg9K=miA@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div>
<ul>
<li>It will be hard to do per-session analyses</li>
</ul>
</div>
</div>
</blockquote>
<br>
I have several responses here:<br>
<br>
1) It will be a bit harder than currently, but I don't think that it
will be extremely hard. There will be an efficient API to fetch all
the pings associated with a user, which should make it relatively
straightforward to stitch together an entire session from its
pieces. This is a functional requirement for the more qualitative
analyses, which will have to stitch together an entire user history
and not just individual sessions. Doing an individual session should
be fairly easy.<br>
<br>
2) I treat the session orientation of telemetry as an unfortunate
limitation, not a desirable property, for almost all of the use
cases that I've seen. I'd like us to try and move away from
reporting metrics based on sessions. Can you describe in more detail
the use cases where analyzing data by session is preferable to
analyzing by some constant denominator? We should be willing to use
both clock time and activeTicks as denominators, and these
denominators can both be calculated looking at individual subsession
pings.<br>
<br>
3) For the case of the current telemetry dashboard, I'd like to
understand why simply replacing the current whole-session analysis
with the new subsessions would produce statistically worse results
than the current session-based analysis.<br>
<br>
<blockquote
cite="mid:CAFsdOVSuZ29a+ePKJbQgrOazkx9YU2La9cvvEALA96Lg9K=miA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<ul>
<li>Many of the 1000+ Telemetry measurements are inherently
"per-session" and can't meaningfully be split into session
fragments:<br>
</li>
<ul>
<li><a moz-do-not-send="true"
href="https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type">Flag
histograms</a> track feature usage per-session.<br>
</li>
<ul>
<li>They are automatically initialized to a value of
"false" at the beginning of a session, and can only be
set to "true" once.<br>
</li>
<li><span style="color:rgb(204,0,0)">If we reset
Telemetry measurements every time we create a new
ping, we'll be reporting nonsense: pings from the
same session will contradict each other on whether a
feature was ever used during the session</span><br>
</li>
<ul>
<li>This would feed bad data to both the dashboards
and any custom analyses<br>
</li>
</ul>
</ul>
</ul>
</ul>
</div>
</div>
</blockquote>
*If* you really care about this per-session, why can't you just take
"true" from any of the subsessions as an indication that it's true
for the entire session?<br>
<br>
And if we just report by subsession, how is this much different from
the skew that we already have between users who have lots of short
sessions and users that keep their browser open for days or weeks?<br>
<br>
Maybe this just indicates that we're mis-using histograms for
non-aggregate measurements, and we should just have a separate list
of flag metrics which are treated differently.<br>
<br>
<blockquote
cite="mid:CAFsdOVSuZ29a+ePKJbQgrOazkx9YU2La9cvvEALA96Lg9K=miA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<ul>
<ul>
<li><a moz-do-not-send="true"
href="https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type">Count
histograms</a> are also per-session measurements. You
can't aggregate a count-histogram value from the middle
of a session together with final values from other
sessions<br>
</li>
</ul>
</ul>
</div>
</div>
</blockquote>
<br>
Won't summing across the subsessions get you the total count for the
session?<br>
<br>
<blockquote
cite="mid:CAFsdOVSuZ29a+ePKJbQgrOazkx9YU2La9cvvEALA96Lg9K=miA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<ul>
<li>For custom analyses, we sometimes want to correlate
measurements from the beginning of a session with
measurements from the end of a session (which could have
lasted several days), e.g. histograms related to startup
performance vs later performance</li>
<ul>
<li>We would need that messy server-side session
reconstruction process to get at per-session data.</li>
<li>More generally, a ping generated as a result of local
time & environment changes is
not inherently meaningful to us, unlike a full user
session</li>
</ul>
</ul>
</div>
</div>
</blockquote>
<br>
I don't understand this case. Assuming session stitching works,
which is a general requirement for all sorts of analyses, this
should work no worse than currently, and you potentially have
finer-grain data on the subsequent days if that's useful.<br>
<br>
<blockquote
cite="mid:CAFsdOVSuZ29a+ePKJbQgrOazkx9YU2La9cvvEALA96Lg9K=miA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<ul>
<li>Resetting Telemetry and FHR data when a
TelemetryExperiment begins removes valuable context from
the experiment ping. It's possible to reconstruct it, but
that's yet another server-side job to run</li>
</ul>
</div>
</div>
</blockquote>
I don't understand this. Is this also assuming that stitching is
expensive?<br>
<br>
<blockquote
cite="mid:CAFsdOVSuZ29a+ePKJbQgrOazkx9YU2La9cvvEALA96Lg9K=miA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<ul>
<li>There's overhead from sending a new ping for each
mid-session environment change</li>
<ul>
<li>There's also a small privacy issue with creating
ordered, fine-grained reports of user actions, e.g. when
a user goes through their add-ons list and disables 5
addons, we report each user action</li>
<li>Either coalesce successive environment-change pings,
or carefully vet which mid-session environment changes
generate a new ping</li>
</ul>
</ul>
</div>
</div>
</blockquote>
<br>
I think it's worth considering whether there's a window of time
where multiple changes get coalesced. But I'm not particularly
worried about the privacy problem, since we do in fact want to
record when users disable addons.<br>
<br>
<blockquote
cite="mid:CAFsdOVSuZ29a+ePKJbQgrOazkx9YU2La9cvvEALA96Lg9K=miA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>I'd like to propose that we implement the following
modifications to the FHR/Telemetry v4 document:<br>
<ol>
<li>Do not reset <u>Telemetry</u> measurements when a
session crosses the 24-hour boundary</li>
<ul>
<li>Continue to "reset" Telemetry measurements when we
start a new session</li>
<li>There's no need to reset Telemetry on most environment
changes (e.g. amount of memory installed) since those
can't happen without a Firefox restart anyway.</li>
</ul>
<li>Record mid-session environment changes (add-ons and
TelemetryExperiments) in a special section in the ping.</li>
<ul>
<li>For each such environment change, document the change
in the section and also attach a snapshot of the
Telemetry & FHR data at the time of the change</li>
<li>After the snapshot is saved, reset Telemetry and FHR
measurements for the current session. In other words,
snapshot & then build up a diff<br>
</li>
<li>For each additional environment change during the same
session, just repeat and append to the new section<br>
</li>
<li>Telemetry backend scripts (dashboard, regression
detector etc) can just ignore experiment/add-on change
pings</li>
</ul>
</ol>
This model has some nice properties:</div>
<div>
<ul>
<li>The <b>final ping</b> of a session is equivalent to a
Telemetry saved-session ping</li>
<ul>
<li>Per-session analyses are as easy to do as before</li>
</ul>
<ul>
<li>No need to run any session reconstruction jobs!</li>
</ul>
<li>Every main ping submitted is meaningful without needing
any reconstruction steps. All pings will contain the
current FHR state + all the Telemetry measurements from
the current session<br>
</li>
<li>Most pings will only have one environment change, so the
relevant measurements that happened after the change are
all going to be in the regular Telemetry/FHR section</li>
<li>However, when deeper analysis is required, Experiment
pings will also have information about what was happening
BEFORE the experiment began</li>
<li>Analyzing pings with multiple environment changes won't
be much harder</li>
</ul>
</div>
</div>
</blockquote>
<br>
I feel like this proposal is optimizing for the wrong things.<br>
<br>
You are making a distinction between "Telemetry" measurements and
other measurements in a way which I am specifically trying to avoid.
The goal is to use the common histogram system for everything. At
least some of those measurements must be distinguished by
subsession. I explicitly want to get rid of the current situation
where "telemetry metrics" are treated one way, and "FHR metrics" are
treated in some entirely separate manner. We want to be able to use
the standard histograms/keyed histograms for almost everything.<br>
<br>
For the simple things like the telemetry dashboard, I believe that
doing all analysis by subsession is good enough (no worse than the
current situation). For more complex queries , both stitching
together an entire session and stitching together the history
per-user will not only be possible but should be fairly efficient.<br>
<br>
--BDS<br>
<br>
</body>
</html>