RFC: New "install" and "update" pings

Wed Mar 8 18:27:39 UTC 2017

On Wed, Mar 08, 2017 at 08:57:45AM -0800, Saptarshi Guha wrote:
> I wanted to chime in.
> 
> 1. I am at least one of the people who want some way to capture
> first run from within UT. We measure downloads, installs and first runs
> from 3 different data sources (GA, stub installer, and GA ). When creating
> reports/analyses these data sources often give 3 varieties of the same
> number. What happens next is that we start pointing blame at data sources
> or ascribing the difference to the data source and leaving it at that. It
> would be streamlined if UT(which is in our control) could contain as much
> as possible. The analysts are very comfortable using UT and having these
> measurements in one data source would keep one thing constant: the data
> source.
> 
> 2. I also advocate (strongly!) doing releases based entirely on UT data. A
> release (and even an experiment) can be released to profiles meeting some
> criteria e.g. Release, geo:US, locale: en-us memory: low-end *and*
> sampleid1 in 1, sampleid2 in 41.
> Here sampleid2 are independent versions of sample_id (e.g.
> crc32(salt(client_id)) %% 100 ). That way we can release to as small
> as 60,000 profiles. Moreover we can keep a file with this release history
> (what filters, what sampleids chosen etc) and keep it in the UT json itself.

These new pings are not going to enable us to target updates to people based on this sort of data, they will simply let us shut off updates when certain targets are reached. We can only target users based on data available in the update URL (eg: OS version, locale, architecture, etc.)

> 
> This echoes what Brendan mentioned before about releases being test driven.
> if we do the above, then we can release  to channel:Release, geo:US,
> locale: en-us memory: low-end *and* sampleid1 in 1, sampleid2 in 41 of
> which 50% in control (no change) and 50% get the new release.
> 
> This abstracts release management and experiments under a common conceptual
> umbrella. Moreover it will make the UT json somewhat self describing.
> 
> 
>  Saptarshi
> 
> 
> On Wed, Mar 8, 2017 at 8:21 AM, Alessio Placitelli <aplacitelli at mozilla.com>
> wrote:
> 
> > + Ben Hearsum
> >
> > Thanks for the good questions, Benjamin! My answers are below.
> >
> > 2017-03-08 15:35 GMT+01:00 Benjamin Smedberg <benjamin at smedbergs.us>:
> >
> >> Can you describe what questions these pings are meant to answer?
> >>
> >>
> > These pings are meant to answer, in "soft real-time", these questions:
> >
> > - How quickly and how many people shift to a new version? e.g. low latency
> > answers to  “how many people are on 53”
> > - How likely is that a user that installs from a given source does more
> > page views?
> > - How does churn relate to the installation source?
> > - Given the discrepancy between download and first run numbers, where is
> > the truth?
> > - How many users are stuck on updates?
> >
> >
> >> For example, one of the use cases for release management is that we want
> >> to be able to roll out new versions progressively to specific numbers of
> >> people. Right now we do this by throttling the updates for a period of
> >> time. It would be faster and better to do a 100% update until we get the
> >> target number of people and then cut it off. But in order to serve that use
> >> case, I don't know whether we need the ping *after* update or the ping
> >> after the update is staged but before it is actually applied.
> >>
> >>
> > To address this scenario, we're considering sending the update ping
> > *before* it attempts to apply it and *after* it's applied.
> >
> >
> >> Similarly for the activation ping, the growth team probably cares most
> >> about new profiles, but release management might care also about people who
> >> don't "update" but "install" the new version.
> >>
> >> Let's say somebody is running Firefox 56 and installs Firefox 57 with the
> >> installer (rather than updater). Would that send *either* of the install or
> >> update pings? Do we need to add a clear signal to the update ping about
> >> whether this is the result of a fresh install or an app update?
> >>
> >>
> > This would send the update ping when Firefox is started. Adding a clear
> > signal about whether this is the result of a fresh install or an app update
> > is a good idea, IMHO.
> >
> >
> >> What about if somebody is running Firefox 56 and reinstalls Firefox 56?
> >> Will that send any pings?
> >>
> >>
> > It shouldn't if the same profile is used.
> >
> >
> >> --BDS
> >>
> >>
> >> On Wed, Mar 8, 2017 at 9:26 AM, Alessio Placitelli <
> >> aplacitelli at mozilla.com> wrote:
> >>
> >>> *tl;dr;* we're kicking off the work on two new ping types, the proposal
> >>> is down below and we're gladly accepting comments.
> >>>
> >>> *Context*
> >>>
> >>> Data Platform will be supporting Firefox Growth by working on reducing
> >>> the data reporting latency, allowing client teams to make decision and
> >>> iterate quickly.
> >>>
> >>> Along with the initiative of sending the "shutdown" ping when Firefox
> >>> shuts down (bug 1336360), we're considering adding the following new pings:
> >>>
> >>> *"Install" (aka "Activation") ping*
> >>> Sent when Firefox is launched right after its installation, on a new
> >>> profile. This ping will contain:
> >>>
> >>>    - The client_id
> >>>    - The environment data (or a subset of the environment data).
> >>>
> >>> Having this clear installation signal would be useful for funnel and
> >>> churn analysis, along with giving us a clearer picture of the "download to
> >>> usage" patterns. The work for this ping is tracked in bug 1120370.
> >>>
> >>> In the current state, the "Install" ping would be sent right after the
> >>> "Data Choices" infobar is displayed, which is 60s into Firefox execution.
> >>> This *could *change in future releases as we will probably be dropping
> >>> the infobar and opening a tab instead.
> >>>
> >>>
> >>> *"Update" ping*Sent right after Firefox upgrades to a new version (i.e.
> >>> restarts after applying the update). It will contain:
> >>>
> >>>    - The client_id
> >>>    - The previous installed version and channel
> >>>    - The current installed version and channel
> >>>
> >>> This will give us a clear signal of a version upgrade, helping to have a
> >>> quicker view on the version uptake of new Firefox releases. The work for
> >>> this ping is tracked in bug 1120372.
> >>>
> >>> There's a discussion going on about sending this ping *before *the
> >>> update takes place and *after *it finishes, to quantify the problems in
> >>> the update process.
> >>> Alternatively, if we were able to detect failed updates when restarting
> >>> Firefox (under investigation), we could just be sending the "update" ping
> >>> once after an update attempt, regardless of its success, together with the
> >>> failure reason.
> >>>
> >>> *Additional context*
> >>> https://docs.google.com/document/d/1NEOHfT4hBsmOmoK5M_wbsF7v
> >>> -NihTtCybHe-sxs9Hgw/edit?ts=58b947d8#heading=h.byypvh46s2wo
> >>>
> >>> _______________________________________________
> >>> fhr-dev mailing list
> >>> fhr-dev at mozilla.org
> >>> https://mail.mozilla.org/listinfo/fhr-dev
> >>>
> >>>
> >>
> >