Decision/Update on Status of Activity Stream on FX57

Jenn Chaulk jchaulk at mozilla.com
Tue Nov 7 14:03:18 UTC 2017


Hello,

Since the last update on October 26th, the Activity Stream team has been
working to finalize a solid contingency plan for FX57, and also analyzing
(with the help of other experts across the org) the data available from
past studies, as well as the 57 Beta Shield Study on Activity Stream
Retention that is currently in flight.

Yesterday the confirmed details of that contingency plan, as well as the
findings from the above mentioned data gathering and analysis were
presented to management for review.  The presentation can be reviewed here
-  Activity Stream: 57 Go/No-Go Decision
<https://docs.google.com/presentation/d/1ynPPRRtwALM8zbNxZRTAyoFFFt2UvBZfd_ANVLCEYuc/edit#slide=id.p>
.

In summary, we had previously seen evidence in 56 Release indicating a drop
in retention for Activity Stream users vs the legacy new tab experience.
Further analysis of the 56 data has suggested that these results are
actually coming from missing data in the experiment dataset, rather than
from an actual drop in retention.  (*Note *- the cause of this missing data
is still unclear.) Also, we were unable to reproduce such a retention drop
while running a similar study in 57 Beta, indicating that Activity Stream
does not actually have any negative impact on user retention in FX57.

Upon conclusion of the presentation the recommendation from management was
to proceed as follows:

   - *Proceed as per original plan: Activity Stream will be rolled out to
   99% of the FX57 population upon release.  A 1% control group will remain on
   the legacy experience.*

The team will continue to meet regularly and review early data from
release, to monitor for any issues or abnormalities.  If any major issues
do occur the team may adjust the release parameters for FX57 (i.e., pull
back to a smaller population percentage) as the situation dictates.

Best Regards,
Jenn

-- 

Jennifer A. Chaulk - PMP
Engineering Program Manager
Mozilla Corporation
Work: 1.416.848.3114 x 3029 <(416)%20848-3114>
Mobile: 519.277.0365 <(519)%20277-0365>
Email: jchaulk at mozilla.com
IRC: jchaulk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20171107/0d277556/attachment.html>


More information about the firefox-dev mailing list