RFC: New "install" and "update" pings

Thu Mar 9 13:48:48 UTC 2017

While we have existing update metrics, reducing the latency in receiving
signals from clients post-install/-update is the main goal here.
Also, having a signal before users can drop off is valuable.

Robert, having you review/feedback the implementation would be valuable to
make sure the update semantics are covered correctly.

Georg

On Wed, Mar 8, 2017 at 9:21 PM, Robert Strong <rstrong at mozilla.com> wrote:

> The UPDATE_STATE_CODE_COMPLETE_STARTUP or UPDATE_STATE_CODE_PARTIAL_STARTUP
> histograms are recorded with a value of 10 on startup after a successful
> update. I don't know how long it is after startup that these (and other)
> histograms are sent though.
>
> Robert
>
>
> On Wed, Mar 8, 2017 at 12:04 PM, Ben Hearsum <bhearsum at mozilla.com> wrote:
>
>> Post-update pings would allow us to do much more accurate throttled
>> rollouts of Firefox updates than we do now, which Release Management has
>> wanted for quite awhile.
>>
>> On Wed, Mar 08, 2017 at 02:08:47PM -0500, David Durst wrote:
>> > We already have lots of telemetry around update -- is there a benefit to
>> > having discrete pings instead?
>> >
>> > --
>> > David Durst [:ddurst]
>> >
>> > On Wed, Mar 8, 2017 at 11:57 AM, Saptarshi Guha <sguha at mozilla.com>
>> wrote:
>> >
>> > > I wanted to chime in.
>> > >
>> > > 1. I am at least one of the people who want some way to capture
>> > > first run from within UT. We measure downloads, installs and first
>> runs
>> > > from 3 different data sources (GA, stub installer, and GA ). When
>> creating
>> > > reports/analyses these data sources often give 3 varieties of the same
>> > > number. What happens next is that we start pointing blame at data
>> sources
>> > > or ascribing the difference to the data source and leaving it at
>> that. It
>> > > would be streamlined if UT(which is in our control) could contain as
>> much
>> > > as possible. The analysts are very comfortable using UT and having
>> these
>> > > measurements in one data source would keep one thing constant: the
>> data
>> > > source.
>> > >
>> > > 2. I also advocate (strongly!) doing releases based entirely on UT
>> data. A
>> > > release (and even an experiment) can be released to profiles meeting
>> some
>> > > criteria e.g. Release, geo:US, locale: en-us memory: low-end *and*
>> > > sampleid1 in 1, sampleid2 in 41.
>> > > Here sampleid2 are independent versions of sample_id (e.g.
>> > > crc32(salt(client_id)) %% 100 ). That way we can release to as small
>> > > as 60,000 profiles. Moreover we can keep a file with this release
>> history
>> > > (what filters, what sampleids chosen etc) and keep it in the UT json
>> itself.
>> > >
>> > > This echoes what Brendan mentioned before about releases being test
>> > > driven. if we do the above, then we can release  to channel:Release,
>> > > geo:US, locale: en-us memory: low-end *and* sampleid1 in 1, sampleid2
>> in 41
>> > > of which 50% in control (no change) and 50% get the new release.
>> > >
>> > > This abstracts release management and experiments under a common
>> > > conceptual umbrella. Moreover it will make the UT json somewhat self
>> > > describing.
>> > >
>> > >
>> > >  Saptarshi
>> > >
>> > >
>> > > On Wed, Mar 8, 2017 at 8:21 AM, Alessio Placitelli <
>> > > aplacitelli at mozilla.com> wrote:
>> > >
>> > >> + Ben Hearsum
>> > >>
>> > >> Thanks for the good questions, Benjamin! My answers are below.
>> > >>
>> > >> 2017-03-08 15:35 GMT+01:00 Benjamin Smedberg <benjamin at smedbergs.us
>> >:
>> > >>
>> > >>> Can you describe what questions these pings are meant to answer?
>> > >>>
>> > >>>
>> > >> These pings are meant to answer, in "soft real-time", these
>> questions:
>> > >>
>> > >> - How quickly and how many people shift to a new version? e.g. low
>> > >> latency answers to  “how many people are on 53”
>> > >> - How likely is that a user that installs from a given source does
>> more
>> > >> page views?
>> > >> - How does churn relate to the installation source?
>> > >> - Given the discrepancy between download and first run numbers,
>> where is
>> > >> the truth?
>> > >> - How many users are stuck on updates?
>> > >>
>> > >>
>> > >>> For example, one of the use cases for release management is that we
>> want
>> > >>> to be able to roll out new versions progressively to specific
>> numbers of
>> > >>> people. Right now we do this by throttling the updates for a period
>> of
>> > >>> time. It would be faster and better to do a 100% update until we
>> get the
>> > >>> target number of people and then cut it off. But in order to serve
>> that use
>> > >>> case, I don't know whether we need the ping *after* update or the
>> ping
>> > >>> after the update is staged but before it is actually applied.
>> > >>>
>> > >>>
>> > >> To address this scenario, we're considering sending the update ping
>> > >> *before* it attempts to apply it and *after* it's applied.
>> > >>
>> > >>
>> > >>> Similarly for the activation ping, the growth team probably cares
>> most
>> > >>> about new profiles, but release management might care also about
>> people who
>> > >>> don't "update" but "install" the new version.
>> > >>>
>> > >>> Let's say somebody is running Firefox 56 and installs Firefox 57
>> with
>> > >>> the installer (rather than updater). Would that send *either* of the
>> > >>> install or update pings? Do we need to add a clear signal to the
>> update
>> > >>> ping about whether this is the result of a fresh install or an app
>> update?
>> > >>>
>> > >>>
>> > >> This would send the update ping when Firefox is started. Adding a
>> clear
>> > >> signal about whether this is the result of a fresh install or an app
>> update
>> > >> is a good idea, IMHO.
>> > >>
>> > >>
>> > >>> What about if somebody is running Firefox 56 and reinstalls Firefox
>> 56?
>> > >>> Will that send any pings?
>> > >>>
>> > >>>
>> > >> It shouldn't if the same profile is used.
>> > >>
>> > >>
>> > >>> --BDS
>> > >>>
>> > >>>
>> > >>> On Wed, Mar 8, 2017 at 9:26 AM, Alessio Placitelli <
>> > >>> aplacitelli at mozilla.com> wrote:
>> > >>>
>> > >>>> *tl;dr;* we're kicking off the work on two new ping types, the
>> > >>>> proposal is down below and we're gladly accepting comments.
>> > >>>>
>> > >>>> *Context*
>> > >>>>
>> > >>>> Data Platform will be supporting Firefox Growth by working on
>> reducing
>> > >>>> the data reporting latency, allowing client teams to make decision
>> and
>> > >>>> iterate quickly.
>> > >>>>
>> > >>>> Along with the initiative of sending the "shutdown" ping when
>> Firefox
>> > >>>> shuts down (bug 1336360), we're considering adding the following
>> new pings:
>> > >>>>
>> > >>>> *"Install" (aka "Activation") ping*
>> > >>>> Sent when Firefox is launched right after its installation, on a
>> new
>> > >>>> profile. This ping will contain:
>> > >>>>
>> > >>>>    - The client_id
>> > >>>>    - The environment data (or a subset of the environment data).
>> > >>>>
>> > >>>> Having this clear installation signal would be useful for funnel
>> and
>> > >>>> churn analysis, along with giving us a clearer picture of the
>> "download to
>> > >>>> usage" patterns. The work for this ping is tracked in bug 1120370.
>> > >>>>
>> > >>>> In the current state, the "Install" ping would be sent right after
>> the
>> > >>>> "Data Choices" infobar is displayed, which is 60s into Firefox
>> execution.
>> > >>>> This *could *change in future releases as we will probably be
>> dropping
>> > >>>> the infobar and opening a tab instead.
>> > >>>>
>> > >>>>
>> > >>>> *"Update" ping*Sent right after Firefox upgrades to a new version
>> > >>>> (i.e. restarts after applying the update). It will contain:
>> > >>>>
>> > >>>>    - The client_id
>> > >>>>    - The previous installed version and channel
>> > >>>>    - The current installed version and channel
>> > >>>>
>> > >>>> This will give us a clear signal of a version upgrade, helping to
>> have
>> > >>>> a quicker view on the version uptake of new Firefox releases. The
>> work for
>> > >>>> this ping is tracked in bug 1120372.
>> > >>>>
>> > >>>> There's a discussion going on about sending this ping *before *the
>> > >>>> update takes place and *after *it finishes, to quantify the
>> problems
>> > >>>> in the update process.
>> > >>>> Alternatively, if we were able to detect failed updates when
>> restarting
>> > >>>> Firefox (under investigation), we could just be sending the
>> "update" ping
>> > >>>> once after an update attempt, regardless of its success, together
>> with the
>> > >>>> failure reason.
>> > >>>>
>> > >>>> *Additional context*
>> > >>>> https://docs.google.com/document/d/1NEOHfT4hBsmOmoK5M_wbsF7v
>> > >>>> -NihTtCybHe-sxs9Hgw/edit?ts=58b947d8#heading=h.byypvh46s2wo
>> > >>>>
>> > >>>> _______________________________________________
>> > >>>> fhr-dev mailing list
>> > >>>> fhr-dev at mozilla.org
>> > >>>> https://mail.mozilla.org/listinfo/fhr-dev
>> > >>>>
>> > >>>>
>> > >>>
>> > >>
>> > >
>> > > _______________________________________________
>> > > fhr-dev mailing list
>> > > fhr-dev at mozilla.org
>> > > https://mail.mozilla.org/listinfo/fhr-dev
>> > >
>> > >
>>
>> _______________________________________________
>> fhr-dev mailing list
>> fhr-dev at mozilla.org
>> https://mail.mozilla.org/listinfo/fhr-dev
>>
>>
>
> _______________________________________________
> fhr-dev mailing list
> fhr-dev at mozilla.org
> https://mail.mozilla.org/listinfo/fhr-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/fhr-dev/attachments/20170309/b06daf24/attachment-0001.html>