RFC: New "install" and "update" pings

Wed Mar 8 20:21:15 UTC 2017

The UPDATE_STATE_CODE_COMPLETE_STARTUP or UPDATE_STATE_CODE_PARTIAL_STARTUP
histograms are recorded with a value of 10 on startup after a successful
update. I don't know how long it is after startup that these (and other)
histograms are sent though.

Robert

On Wed, Mar 8, 2017 at 12:04 PM, Ben Hearsum <bhearsum at mozilla.com> wrote:

> Post-update pings would allow us to do much more accurate throttled
> rollouts of Firefox updates than we do now, which Release Management has
> wanted for quite awhile.
>
> On Wed, Mar 08, 2017 at 02:08:47PM -0500, David Durst wrote:
> > We already have lots of telemetry around update -- is there a benefit to
> > having discrete pings instead?
> >
> > --
> > David Durst [:ddurst]
> >
> > On Wed, Mar 8, 2017 at 11:57 AM, Saptarshi Guha <sguha at mozilla.com>
> wrote:
> >
> > > I wanted to chime in.
> > >
> > > 1. I am at least one of the people who want some way to capture
> > > first run from within UT. We measure downloads, installs and first runs
> > > from 3 different data sources (GA, stub installer, and GA ). When
> creating
> > > reports/analyses these data sources often give 3 varieties of the same
> > > number. What happens next is that we start pointing blame at data
> sources
> > > or ascribing the difference to the data source and leaving it at that.
> It
> > > would be streamlined if UT(which is in our control) could contain as
> much
> > > as possible. The analysts are very comfortable using UT and having
> these
> > > measurements in one data source would keep one thing constant: the data
> > > source.
> > >
> > > 2. I also advocate (strongly!) doing releases based entirely on UT
> data. A
> > > release (and even an experiment) can be released to profiles meeting
> some
> > > criteria e.g. Release, geo:US, locale: en-us memory: low-end *and*
> > > sampleid1 in 1, sampleid2 in 41.
> > > Here sampleid2 are independent versions of sample_id (e.g.
> > > crc32(salt(client_id)) %% 100 ). That way we can release to as small
> > > as 60,000 profiles. Moreover we can keep a file with this release
> history
> > > (what filters, what sampleids chosen etc) and keep it in the UT json
> itself.
> > >
> > > This echoes what Brendan mentioned before about releases being test
> > > driven. if we do the above, then we can release  to channel:Release,
> > > geo:US, locale: en-us memory: low-end *and* sampleid1 in 1, sampleid2
> in 41
> > > of which 50% in control (no change) and 50% get the new release.
> > >
> > > This abstracts release management and experiments under a common
> > > conceptual umbrella. Moreover it will make the UT json somewhat self
> > > describing.
> > >
> > >
> > >  Saptarshi
> > >
> > >
> > > On Wed, Mar 8, 2017 at 8:21 AM, Alessio Placitelli <
> > > aplacitelli at mozilla.com> wrote:
> > >
> > >> + Ben Hearsum
> > >>
> > >> Thanks for the good questions, Benjamin! My answers are below.
> > >>
> > >> 2017-03-08 15:35 GMT+01:00 Benjamin Smedberg <benjamin at smedbergs.us>:
> > >>
> > >>> Can you describe what questions these pings are meant to answer?
> > >>>
> > >>>
> > >> These pings are meant to answer, in "soft real-time", these questions:
> > >>
> > >> - How quickly and how many people shift to a new version? e.g. low
> > >> latency answers to  “how many people are on 53”
> > >> - How likely is that a user that installs from a given source does
> more
> > >> page views?
> > >> - How does churn relate to the installation source?
> > >> - Given the discrepancy between download and first run numbers, where
> is
> > >> the truth?
> > >> - How many users are stuck on updates?
> > >>
> > >>
> > >>> For example, one of the use cases for release management is that we
> want
> > >>> to be able to roll out new versions progressively to specific
> numbers of
> > >>> people. Right now we do this by throttling the updates for a period
> of
> > >>> time. It would be faster and better to do a 100% update until we get
> the
> > >>> target number of people and then cut it off. But in order to serve
> that use
> > >>> case, I don't know whether we need the ping *after* update or the
> ping
> > >>> after the update is staged but before it is actually applied.
> > >>>
> > >>>
> > >> To address this scenario, we're considering sending the update ping
> > >> *before* it attempts to apply it and *after* it's applied.
> > >>
> > >>
> > >>> Similarly for the activation ping, the growth team probably cares
> most
> > >>> about new profiles, but release management might care also about
> people who
> > >>> don't "update" but "install" the new version.
> > >>>
> > >>> Let's say somebody is running Firefox 56 and installs Firefox 57 with
> > >>> the installer (rather than updater). Would that send *either* of the
> > >>> install or update pings? Do we need to add a clear signal to the
> update
> > >>> ping about whether this is the result of a fresh install or an app
> update?
> > >>>
> > >>>
> > >> This would send the update ping when Firefox is started. Adding a
> clear
> > >> signal about whether this is the result of a fresh install or an app
> update
> > >> is a good idea, IMHO.
> > >>
> > >>
> > >>> What about if somebody is running Firefox 56 and reinstalls Firefox
> 56?
> > >>> Will that send any pings?
> > >>>
> > >>>
> > >> It shouldn't if the same profile is used.
> > >>
> > >>
> > >>> --BDS
> > >>>
> > >>>
> > >>> On Wed, Mar 8, 2017 at 9:26 AM, Alessio Placitelli <
> > >>> aplacitelli at mozilla.com> wrote:
> > >>>
> > >>>> *tl;dr;* we're kicking off the work on two new ping types, the
> > >>>> proposal is down below and we're gladly accepting comments.
> > >>>>
> > >>>> *Context*
> > >>>>
> > >>>> Data Platform will be supporting Firefox Growth by working on
> reducing
> > >>>> the data reporting latency, allowing client teams to make decision
> and
> > >>>> iterate quickly.
> > >>>>
> > >>>> Along with the initiative of sending the "shutdown" ping when
> Firefox
> > >>>> shuts down (bug 1336360), we're considering adding the following
> new pings:
> > >>>>
> > >>>> *"Install" (aka "Activation") ping*
> > >>>> Sent when Firefox is launched right after its installation, on a new
> > >>>> profile. This ping will contain:
> > >>>>
> > >>>>    - The client_id
> > >>>>    - The environment data (or a subset of the environment data).
> > >>>>
> > >>>> Having this clear installation signal would be useful for funnel and
> > >>>> churn analysis, along with giving us a clearer picture of the
> "download to
> > >>>> usage" patterns. The work for this ping is tracked in bug 1120370.
> > >>>>
> > >>>> In the current state, the "Install" ping would be sent right after
> the
> > >>>> "Data Choices" infobar is displayed, which is 60s into Firefox
> execution.
> > >>>> This *could *change in future releases as we will probably be
> dropping
> > >>>> the infobar and opening a tab instead.
> > >>>>
> > >>>>
> > >>>> *"Update" ping*Sent right after Firefox upgrades to a new version
> > >>>> (i.e. restarts after applying the update). It will contain:
> > >>>>
> > >>>>    - The client_id
> > >>>>    - The previous installed version and channel
> > >>>>    - The current installed version and channel
> > >>>>
> > >>>> This will give us a clear signal of a version upgrade, helping to
> have
> > >>>> a quicker view on the version uptake of new Firefox releases. The
> work for
> > >>>> this ping is tracked in bug 1120372.
> > >>>>
> > >>>> There's a discussion going on about sending this ping *before *the
> > >>>> update takes place and *after *it finishes, to quantify the problems
> > >>>> in the update process.
> > >>>> Alternatively, if we were able to detect failed updates when
> restarting
> > >>>> Firefox (under investigation), we could just be sending the
> "update" ping
> > >>>> once after an update attempt, regardless of its success, together
> with the
> > >>>> failure reason.
> > >>>>
> > >>>> *Additional context*
> > >>>> https://docs.google.com/document/d/1NEOHfT4hBsmOmoK5M_wbsF7v
> > >>>> -NihTtCybHe-sxs9Hgw/edit?ts=58b947d8#heading=h.byypvh46s2wo
> > >>>>
> > >>>> _______________________________________________
> > >>>> fhr-dev mailing list
> > >>>> fhr-dev at mozilla.org
> > >>>> https://mail.mozilla.org/listinfo/fhr-dev
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> > > _______________________________________________
> > > fhr-dev mailing list
> > > fhr-dev at mozilla.org
> > > https://mail.mozilla.org/listinfo/fhr-dev
> > >
> > >
>
> _______________________________________________
> fhr-dev mailing list
> fhr-dev at mozilla.org
> https://mail.mozilla.org/listinfo/fhr-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/fhr-dev/attachments/20170308/9d224f9c/attachment-0001.html>