RFC: New "install" and "update" pings

Robert Strong rstrong at mozilla.com
Thu Mar 9 22:23:34 UTC 2017 

Not a problem Georg and I already commented in the bug regarding the post
update ping.

I'm not sure if a pre update ping would be all that useful without
including each step of the update process, etc. You can see a subset of
what is collected in the Firefox Application Update Out Of Date Dashboard
https://telemetry.mozilla.org/update-orphaning/

Cheers,
Robert


On Thu, Mar 9, 2017 at 5:48 AM, Georg Fritzsche <gfritzsche at mozilla.com>
wrote:

> While we have existing update metrics, reducing the latency in receiving
> signals from clients post-install/-update is the main goal here.
> Also, having a signal before users can drop off is valuable.
>
> Robert, having you review/feedback the implementation would be valuable to
> make sure the update semantics are covered correctly.
>
> Georg
>
> On Wed, Mar 8, 2017 at 9:21 PM, Robert Strong <rstrong at mozilla.com> wrote:
>
>> The UPDATE_STATE_CODE_COMPLETE_STARTUP or UPDATE_STATE_CODE_PARTIAL_STARTUP
>> histograms are recorded with a value of 10 on startup after a successful
>> update. I don't know how long it is after startup that these (and other)
>> histograms are sent though.
>>
>> Robert
>>
>>
>> On Wed, Mar 8, 2017 at 12:04 PM, Ben Hearsum <bhearsum at mozilla.com>
>> wrote:
>>
>>> Post-update pings would allow us to do much more accurate throttled
>>> rollouts of Firefox updates than we do now, which Release Management has
>>> wanted for quite awhile.
>>>
>>> On Wed, Mar 08, 2017 at 02:08:47PM -0500, David Durst wrote:
>>> > We already have lots of telemetry around update -- is there a benefit
>>> to
>>> > having discrete pings instead?
>>> >
>>> > --
>>> > David Durst [:ddurst]
>>> >
>>> > On Wed, Mar 8, 2017 at 11:57 AM, Saptarshi Guha <sguha at mozilla.com>
>>> wrote:
>>> >
>>> > > I wanted to chime in.
>>> > >
>>> > > 1. I am at least one of the people who want some way to capture
>>> > > first run from within UT. We measure downloads, installs and first
>>> runs
>>> > > from 3 different data sources (GA, stub installer, and GA ). When
>>> creating
>>> > > reports/analyses these data sources often give 3 varieties of the
>>> same
>>> > > number. What happens next is that we start pointing blame at data
>>> sources
>>> > > or ascribing the difference to the data source and leaving it at
>>> that. It
>>> > > would be streamlined if UT(which is in our control) could contain as
>>> much
>>> > > as possible. The analysts are very comfortable using UT and having
>>> these
>>> > > measurements in one data source would keep one thing constant: the
>>> data
>>> > > source.
>>> > >
>>> > > 2. I also advocate (strongly!) doing releases based entirely on UT
>>> data. A
>>> > > release (and even an experiment) can be released to profiles meeting
>>> some
>>> > > criteria e.g. Release, geo:US, locale: en-us memory: low-end *and*
>>> > > sampleid1 in 1, sampleid2 in 41.
>>> > > Here sampleid2 are independent versions of sample_id (e.g.
>>> > > crc32(salt(client_id)) %% 100 ). That way we can release to as small
>>> > > as 60,000 profiles. Moreover we can keep a file with this release
>>> history
>>> > > (what filters, what sampleids chosen etc) and keep it in the UT json
>>> itself.
>>> > >
>>> > > This echoes what Brendan mentioned before about releases being test
>>> > > driven. if we do the above, then we can release  to channel:Release,
>>> > > geo:US, locale: en-us memory: low-end *and* sampleid1 in 1,
>>> sampleid2 in 41
>>> > > of which 50% in control (no change) and 50% get the new release.
>>> > >
>>> > > This abstracts release management and experiments under a common
>>> > > conceptual umbrella. Moreover it will make the UT json somewhat self
>>> > > describing.
>>> > >
>>> > >
>>> > >  Saptarshi
>>> > >
>>> > >
>>> > > On Wed, Mar 8, 2017 at 8:21 AM, Alessio Placitelli <
>>> > > aplacitelli at mozilla.com> wrote:
>>> > >
>>> > >> + Ben Hearsum
>>> > >>
>>> > >> Thanks for the good questions, Benjamin! My answers are below.
>>> > >>
>>> > >> 2017-03-08 15:35 GMT+01:00 Benjamin Smedberg <benjamin at smedbergs.us
>>> >:
>>> > >>
>>> > >>> Can you describe what questions these pings are meant to answer?
>>> > >>>
>>> > >>>
>>> > >> These pings are meant to answer, in "soft real-time", these
>>> questions:
>>> > >>
>>> > >> - How quickly and how many people shift to a new version? e.g. low
>>> > >> latency answers to  “how many people are on 53”
>>> > >> - How likely is that a user that installs from a given source does
>>> more
>>> > >> page views?
>>> > >> - How does churn relate to the installation source?
>>> > >> - Given the discrepancy between download and first run numbers,
>>> where is
>>> > >> the truth?
>>> > >> - How many users are stuck on updates?
>>> > >>
>>> > >>
>>> > >>> For example, one of the use cases for release management is that
>>> we want
>>> > >>> to be able to roll out new versions progressively to specific
>>> numbers of
>>> > >>> people. Right now we do this by throttling the updates for a
>>> period of
>>> > >>> time. It would be faster and better to do a 100% update until we
>>> get the
>>> > >>> target number of people and then cut it off. But in order to serve
>>> that use
>>> > >>> case, I don't know whether we need the ping *after* update or the
>>> ping
>>> > >>> after the update is staged but before it is actually applied.
>>> > >>>
>>> > >>>
>>> > >> To address this scenario, we're considering sending the update ping
>>> > >> *before* it attempts to apply it and *after* it's applied.
>>> > >>
>>> > >>
>>> > >>> Similarly for the activation ping, the growth team probably cares
>>> most
>>> > >>> about new profiles, but release management might care also about
>>> people who
>>> > >>> don't "update" but "install" the new version.
>>> > >>>
>>> > >>> Let's say somebody is running Firefox 56 and installs Firefox 57
>>> with
>>> > >>> the installer (rather than updater). Would that send *either* of
>>> the
>>> > >>> install or update pings? Do we need to add a clear signal to the
>>> update
>>> > >>> ping about whether this is the result of a fresh install or an app
>>> update?
>>> > >>>
>>> > >>>
>>> > >> This would send the update ping when Firefox is started. Adding a
>>> clear
>>> > >> signal about whether this is the result of a fresh install or an
>>> app update
>>> > >> is a good idea, IMHO.
>>> > >>
>>> > >>
>>> > >>> What about if somebody is running Firefox 56 and reinstalls
>>> Firefox 56?
>>> > >>> Will that send any pings?
>>> > >>>
>>> > >>>
>>> > >> It shouldn't if the same profile is used.
>>> > >>
>>> > >>
>>> > >>> --BDS
>>> > >>>
>>> > >>>
>>> > >>> On Wed, Mar 8, 2017 at 9:26 AM, Alessio Placitelli <
>>> > >>> aplacitelli at mozilla.com> wrote:
>>> > >>>
>>> > >>>> *tl;dr;* we're kicking off the work on two new ping types, the
>>> > >>>> proposal is down below and we're gladly accepting comments.
>>> > >>>>
>>> > >>>> *Context*
>>> > >>>>
>>> > >>>> Data Platform will be supporting Firefox Growth by working on
>>> reducing
>>> > >>>> the data reporting latency, allowing client teams to make
>>> decision and
>>> > >>>> iterate quickly.
>>> > >>>>
>>> > >>>> Along with the initiative of sending the "shutdown" ping when
>>> Firefox
>>> > >>>> shuts down (bug 1336360), we're considering adding the following
>>> new pings:
>>> > >>>>
>>> > >>>> *"Install" (aka "Activation") ping*
>>> > >>>> Sent when Firefox is launched right after its installation, on a
>>> new
>>> > >>>> profile. This ping will contain:
>>> > >>>>
>>> > >>>>    - The client_id
>>> > >>>>    - The environment data (or a subset of the environment data).
>>> > >>>>
>>> > >>>> Having this clear installation signal would be useful for funnel
>>> and
>>> > >>>> churn analysis, along with giving us a clearer picture of the
>>> "download to
>>> > >>>> usage" patterns. The work for this ping is tracked in bug 1120370.
>>> > >>>>
>>> > >>>> In the current state, the "Install" ping would be sent right
>>> after the
>>> > >>>> "Data Choices" infobar is displayed, which is 60s into Firefox
>>> execution.
>>> > >>>> This *could *change in future releases as we will probably be
>>> dropping
>>> > >>>> the infobar and opening a tab instead.
>>> > >>>>
>>> > >>>>
>>> > >>>> *"Update" ping*Sent right after Firefox upgrades to a new version
>>> > >>>> (i.e. restarts after applying the update). It will contain:
>>> > >>>>
>>> > >>>>    - The client_id
>>> > >>>>    - The previous installed version and channel
>>> > >>>>    - The current installed version and channel
>>> > >>>>
>>> > >>>> This will give us a clear signal of a version upgrade, helping to
>>> have
>>> > >>>> a quicker view on the version uptake of new Firefox releases. The
>>> work for
>>> > >>>> this ping is tracked in bug 1120372.
>>> > >>>>
>>> > >>>> There's a discussion going on about sending this ping *before *the
>>> > >>>> update takes place and *after *it finishes, to quantify the
>>> problems
>>> > >>>> in the update process.
>>> > >>>> Alternatively, if we were able to detect failed updates when
>>> restarting
>>> > >>>> Firefox (under investigation), we could just be sending the
>>> "update" ping
>>> > >>>> once after an update attempt, regardless of its success, together
>>> with the
>>> > >>>> failure reason.
>>> > >>>>
>>> > >>>> *Additional context*
>>> > >>>> https://docs.google.com/document/d/1NEOHfT4hBsmOmoK5M_wbsF7v
>>> > >>>> -NihTtCybHe-sxs9Hgw/edit?ts=58b947d8#heading=h.byypvh46s2wo
>>> > >>>>
>>> > >>>> _______________________________________________
>>> > >>>> fhr-dev mailing list
>>> > >>>> fhr-dev at mozilla.org
>>> > >>>> https://mail.mozilla.org/listinfo/fhr-dev
>>> > >>>>
>>> > >>>>
>>> > >>>
>>> > >>
>>> > >
>>> > > _______________________________________________
>>> > > fhr-dev mailing list
>>> > > fhr-dev at mozilla.org
>>> > > https://mail.mozilla.org/listinfo/fhr-dev
>>> > >
>>> > >
>>>
>>> _______________________________________________
>>> fhr-dev mailing list
>>> fhr-dev at mozilla.org
>>> https://mail.mozilla.org/listinfo/fhr-dev
>>>
>>>
>>
>> _______________________________________________
>> fhr-dev mailing list
>> fhr-dev at mozilla.org
>> https://mail.mozilla.org/listinfo/fhr-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/fhr-dev/attachments/20170309/2b3d00f9/attachment.html>

More information about the fhr-dev mailing list