Decision/Update on Status of Activity Stream on FX57

Ritu Kothari rkothari at mozilla.com
Tue Nov 7 19:40:06 UTC 2017


On Tue, Nov 7, 2017 at 6:03 AM, Jenn Chaulk <jchaulk at mozilla.com> wrote:

> Hello,
>
> Since the last update on October 26th, the Activity Stream team has been
> working to finalize a solid contingency plan for FX57, and also analyzing
> (with the help of other experts across the org) the data available from
> past studies, as well as the 57 Beta Shield Study on Activity Stream
> Retention that is currently in flight.
>
> Yesterday the confirmed details of that contingency plan, as well as the
> findings from the above mentioned data gathering and analysis were
> presented to management for review.  The presentation can be reviewed here
> -  Activity Stream: 57 Go/No-Go Decision
> <https://docs.google.com/presentation/d/1ynPPRRtwALM8zbNxZRTAyoFFFt2UvBZfd_ANVLCEYuc/edit#slide=id.p>
> .
>
> In summary, we had previously seen evidence in 56 Release indicating a
> drop in retention for Activity Stream users vs the legacy new tab
> experience.  Further analysis of the 56 data has suggested that these
> results are actually coming from missing data in the experiment dataset,
> rather than from an actual drop in retention.  (*Note *- the cause of
> this missing data is still unclear.) Also, we were unable to reproduce such
> a retention drop while running a similar study in 57 Beta, indicating that
> Activity Stream does not actually have any negative impact on user
> retention in FX57.
>
> Upon conclusion of the presentation the recommendation from management was
> to proceed as follows:
>
>    - *Proceed as per original plan: Activity Stream will be rolled out to
>    99% of the FX57 population upon release.  A 1% control group will remain on
>    the legacy experience.*
>
> Thanks for your email Jenn. In the past, when we've had a GO decision on
features and it is a pref controlled feature, we typically go live with the
pref turned ON from the moment we launch i.e. 100% rollout. In this case,
are we planning to go-live with the pref for AS set to ON and planning to
run a SHIELD study that turns AS off for 1% of release population?

Thanks,
Ritu

The team will continue to meet regularly and review early data from
> release, to monitor for any issues or abnormalities.  If any major issues
> do occur the team may adjust the release parameters for FX57 (i.e., pull
> back to a smaller population percentage) as the situation dictates.
>
> Best Regards,
> Jenn
>
> --
>
> Jennifer A. Chaulk - PMP
> Engineering Program Manager
> Mozilla Corporation
> Work: 1.416.848.3114 x 3029 <(416)%20848-3114>
> Mobile: 519.277.0365 <(519)%20277-0365>
> Email: jchaulk at mozilla.com
> IRC: jchaulk
>
> --
> You received this message because you are subscribed to the Google Groups
> "epm-status" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to epm-status+unsubscribe at mozilla.com.
> To post to this group, send email to epm-status at mozilla.com.
> To view this discussion on the web visit https://groups.google.com/a/
> mozilla.com/d/msgid/epm-status/CAOxuPHExAVpnGKXfbNWrRJdMGoJe6
> CUYiev04tB8Bcp8Lv%3DSrw%40mail.gmail.com
> <https://groups.google.com/a/mozilla.com/d/msgid/epm-status/CAOxuPHExAVpnGKXfbNWrRJdMGoJe6CUYiev04tB8Bcp8Lv%3DSrw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20171107/1044c74f/attachment.html>


More information about the firefox-dev mailing list