Suggestions for the new unified FHR/Telemetry/Experiment ping

[Brendan Colloran]

A few thoughts:

* Thanks for these detailed comments Vlad, I'm happy to see this is
all being carefully vetted.

* Session/day stitching per user has been stated as a *hard
requirement* for FHR since early in this project -- we need to be able
to access data in a form that looks basically like the monolithic FHR
packets per user that we collect today.  We've been assured that this
is will not be a problem, but if you have reason to believe that it
will be infeasible, that is of concern to the metrics team as well.

* Local midnight is the correct break point between days for FHR,
because that is the notion of day that maps into the diurnal behavior
of most users.

* Vlad, you mention "A single missing or corrupted fragment in a long
session means we won't be able to recreate the session. We already
have problems with missing clientIDs". This is the first I've heard of
missing clientIDs, please elaborate. After all pain of the FHR
orphaning cluster... if clientID submission isn't *bulletproof*,
that's a show stopper for FHR, because without monolithic payloads, it
means orphaned session fragments (aka data loss).

* If I'm reading you proposal correctly, you're not proposing any
changes to the data that will be collected by FHR (or telemetry?) or
any changes to the way FHR data will be binned and aggregated-- you're
just advocating for a more conservative roll out plan, yes?

-bc

On Tue, Jan 27, 2015 at 8:59 AM, Georg Fritzsche <gfritzsche at mozilla.com> wrote:
>
> On 27 Jan 2015, at 07:11, Vladan Djeric <vdjeric at mozilla.com> wrote:
>>
>> In the new system, Firefox starts collecting a new ping whenever:
>>
>> a new Firefox session is started
>> a new day has begun (not sure if it's every 24 hours of uptime, or if it's
>> based on local time e.g. midnight local time)
>>
>> That was a partially open question, midnight local time is what we will do
>> unless there is useful counter arguments.
>
>
> If FHR analyses are based around local time, we have to start pings on
> midnight local time. This raises some challenges for how we store the
> submitted pings server-side. The S3 archives are currently organized by
> submission date not (local) collection date. This discussion is probably
> better to have in person with Mark.
>
>
> If the splitting of the session at that point is a problem, let me know - i
> don’t know what the server-side requirements for organisation are.
> We do need to split at some point for the 24h submission interval, midnight
> local time was brought up as a probably convenient cut-time.
> What exact time we choose shouldn’t matter too much?
>
> Georg
>
> _______________________________________________
> fhr-dev mailing list
> fhr-dev at mozilla.org
> https://mail.mozilla.org/listinfo/fhr-dev
>