Heavy Users Dataset - Available Now!

[Rebecca Weiss]

AMAZING.  Thanks Frank!

On Thu, Oct 5, 2017 at 9:22 AM, Saptarshi Guha <sguha at mozilla.com> wrote:

> Lovely news. Thanks for your efforts on this
> Regards
> Saptarshi
>
> On Thu, Oct 5, 2017 at 12:18 AM, Alessio Placitelli <
> aplacitelli at mozilla.com> wrote:
>
>> Great job Frank!
>>
>> 2017-10-05 5:50 GMT+02:00 Frank Bertsch <fbertsch at mozilla.com>:
>>
>>> All -
>>>
>>> After much weeping, wailing, and gnashing of teeth, the heavy_users
>>> dataset is available. It is one-row per client-day, where day is
>>> submission_date. A client has a row for a specific submission_date if they
>>> were active at all in the 28 day window ending on that submission_date.
>>> How to use it:
>>>
>>> In Spark: `spark.read.parquet("s3://telemetry-parquet/heavy_users/v1")
>>>
>>> In SQL: "SELECT * FROM heavy_users"
>>>
>>> Below are a few example queries for you to peruse.
>>>
>>> https://sql.telemetry.mozilla.org/queries/47041/source#127382 - *Joined
>>> with main_summary* to get distribution of max_concurrent_tab_count for
>>> heavy vs. non-heavy users
>>>
>>> https://sql.telemetry.mozilla.org/queries/47044/source#127385 - *Joined
>>> with longitudinal* to get crash rates for heavy vs. non-heavy users
>>>
>>> You'll note that it seems that heavy_users use more tabs, but crash
>>> less. These results probably require more investigation.
>>> What is it?
>>>
>>> A user is a heavy_user as of day N if, for the 28 day period ending on
>>> day N, the sum of their active_ticks is in the 90th percentile (or above)
>>> of all clients during that period. For more analysis on this, and a
>>> discussion of new profiles, see https://metrics.mozilla.com/pr
>>> otected/sguha/heavy/heavycutoffs5.html.
>>>
>>> Please note a few caveats:
>>>
>>> 0. Data starts at 20170801. There is technically data in the table
>>> before this, but heavy_user is NULL for those dates because it needed to
>>> bootstrap the first 28 day window.
>>>
>>> 1. Because it is top 10% for each 28 day period, and single
>>> submission_date has more than 10% of clients be considered heavy_users.
>>> This is because heavy_users, on average, use Firefox on more days than
>>> non-heavy users.
>>>
>>> 2. Each day has a separate, but related, set of heavy_users. Initial
>>> investigations show that ~97.5% of heavy_user as of a certain day are still
>>> considered heavy_users as of the next day.
>>>
>>> -Frank
>>>
>>> _______________________________________________
>>> Fx-data-dev mailing list
>>> Fx-data-dev at mozilla.org
>>> https://mail.mozilla.org/listinfo/fx-data-dev
>>>
>>>
>>
>> _______________________________________________
>> Fx-data-dev mailing list
>> Fx-data-dev at mozilla.org
>> https://mail.mozilla.org/listinfo/fx-data-dev
>>
>>
>
> _______________________________________________
> Fx-data-dev mailing list
> Fx-data-dev at mozilla.org
> https://mail.mozilla.org/listinfo/fx-data-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/fx-data-dev/attachments/20171005/f2707f1a/attachment.html>