RFC: New "install" and "update" pings

Wed Mar 8 19:08:47 UTC 2017

We already have lots of telemetry around update -- is there a benefit to
having discrete pings instead?

--
David Durst [:ddurst]

On Wed, Mar 8, 2017 at 11:57 AM, Saptarshi Guha <sguha at mozilla.com> wrote:

> I wanted to chime in.
>
> 1. I am at least one of the people who want some way to capture
> first run from within UT. We measure downloads, installs and first runs
> from 3 different data sources (GA, stub installer, and GA ). When creating
> reports/analyses these data sources often give 3 varieties of the same
> number. What happens next is that we start pointing blame at data sources
> or ascribing the difference to the data source and leaving it at that. It
> would be streamlined if UT(which is in our control) could contain as much
> as possible. The analysts are very comfortable using UT and having these
> measurements in one data source would keep one thing constant: the data
> source.
>
> 2. I also advocate (strongly!) doing releases based entirely on UT data. A
> release (and even an experiment) can be released to profiles meeting some
> criteria e.g. Release, geo:US, locale: en-us memory: low-end *and*
> sampleid1 in 1, sampleid2 in 41.
> Here sampleid2 are independent versions of sample_id (e.g.
> crc32(salt(client_id)) %% 100 ). That way we can release to as small
> as 60,000 profiles. Moreover we can keep a file with this release history
> (what filters, what sampleids chosen etc) and keep it in the UT json itself.
>
> This echoes what Brendan mentioned before about releases being test
> driven. if we do the above, then we can release  to channel:Release,
> geo:US, locale: en-us memory: low-end *and* sampleid1 in 1, sampleid2 in 41
> of which 50% in control (no change) and 50% get the new release.
>
> This abstracts release management and experiments under a common
> conceptual umbrella. Moreover it will make the UT json somewhat self
> describing.
>
>
>  Saptarshi
>
>
> On Wed, Mar 8, 2017 at 8:21 AM, Alessio Placitelli <
> aplacitelli at mozilla.com> wrote:
>
>> + Ben Hearsum
>>
>> Thanks for the good questions, Benjamin! My answers are below.
>>
>> 2017-03-08 15:35 GMT+01:00 Benjamin Smedberg <benjamin at smedbergs.us>:
>>
>>> Can you describe what questions these pings are meant to answer?
>>>
>>>
>> These pings are meant to answer, in "soft real-time", these questions:
>>
>> - How quickly and how many people shift to a new version? e.g. low
>> latency answers to  “how many people are on 53”
>> - How likely is that a user that installs from a given source does more
>> page views?
>> - How does churn relate to the installation source?
>> - Given the discrepancy between download and first run numbers, where is
>> the truth?
>> - How many users are stuck on updates?
>>
>>
>>> For example, one of the use cases for release management is that we want
>>> to be able to roll out new versions progressively to specific numbers of
>>> people. Right now we do this by throttling the updates for a period of
>>> time. It would be faster and better to do a 100% update until we get the
>>> target number of people and then cut it off. But in order to serve that use
>>> case, I don't know whether we need the ping *after* update or the ping
>>> after the update is staged but before it is actually applied.
>>>
>>>
>> To address this scenario, we're considering sending the update ping
>> *before* it attempts to apply it and *after* it's applied.
>>
>>
>>> Similarly for the activation ping, the growth team probably cares most
>>> about new profiles, but release management might care also about people who
>>> don't "update" but "install" the new version.
>>>
>>> Let's say somebody is running Firefox 56 and installs Firefox 57 with
>>> the installer (rather than updater). Would that send *either* of the
>>> install or update pings? Do we need to add a clear signal to the update
>>> ping about whether this is the result of a fresh install or an app update?
>>>
>>>
>> This would send the update ping when Firefox is started. Adding a clear
>> signal about whether this is the result of a fresh install or an app update
>> is a good idea, IMHO.
>>
>>
>>> What about if somebody is running Firefox 56 and reinstalls Firefox 56?
>>> Will that send any pings?
>>>
>>>
>> It shouldn't if the same profile is used.
>>
>>
>>> --BDS
>>>
>>>
>>> On Wed, Mar 8, 2017 at 9:26 AM, Alessio Placitelli <
>>> aplacitelli at mozilla.com> wrote:
>>>
>>>> *tl;dr;* we're kicking off the work on two new ping types, the
>>>> proposal is down below and we're gladly accepting comments.
>>>>
>>>> *Context*
>>>>
>>>> Data Platform will be supporting Firefox Growth by working on reducing
>>>> the data reporting latency, allowing client teams to make decision and
>>>> iterate quickly.
>>>>
>>>> Along with the initiative of sending the "shutdown" ping when Firefox
>>>> shuts down (bug 1336360), we're considering adding the following new pings:
>>>>
>>>> *"Install" (aka "Activation") ping*
>>>> Sent when Firefox is launched right after its installation, on a new
>>>> profile. This ping will contain:
>>>>
>>>>    - The client_id
>>>>    - The environment data (or a subset of the environment data).
>>>>
>>>> Having this clear installation signal would be useful for funnel and
>>>> churn analysis, along with giving us a clearer picture of the "download to
>>>> usage" patterns. The work for this ping is tracked in bug 1120370.
>>>>
>>>> In the current state, the "Install" ping would be sent right after the
>>>> "Data Choices" infobar is displayed, which is 60s into Firefox execution.
>>>> This *could *change in future releases as we will probably be dropping
>>>> the infobar and opening a tab instead.
>>>>
>>>>
>>>> *"Update" ping*Sent right after Firefox upgrades to a new version
>>>> (i.e. restarts after applying the update). It will contain:
>>>>
>>>>    - The client_id
>>>>    - The previous installed version and channel
>>>>    - The current installed version and channel
>>>>
>>>> This will give us a clear signal of a version upgrade, helping to have
>>>> a quicker view on the version uptake of new Firefox releases. The work for
>>>> this ping is tracked in bug 1120372.
>>>>
>>>> There's a discussion going on about sending this ping *before *the
>>>> update takes place and *after *it finishes, to quantify the problems
>>>> in the update process.
>>>> Alternatively, if we were able to detect failed updates when restarting
>>>> Firefox (under investigation), we could just be sending the "update" ping
>>>> once after an update attempt, regardless of its success, together with the
>>>> failure reason.
>>>>
>>>> *Additional context*
>>>> https://docs.google.com/document/d/1NEOHfT4hBsmOmoK5M_wbsF7v
>>>> -NihTtCybHe-sxs9Hgw/edit?ts=58b947d8#heading=h.byypvh46s2wo
>>>>
>>>> _______________________________________________
>>>> fhr-dev mailing list
>>>> fhr-dev at mozilla.org
>>>> https://mail.mozilla.org/listinfo/fhr-dev
>>>>
>>>>
>>>
>>
>
> _______________________________________________
> fhr-dev mailing list
> fhr-dev at mozilla.org
> https://mail.mozilla.org/listinfo/fhr-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/fhr-dev/attachments/20170308/91c56c89/attachment.html>