<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">We have only started testing the Activity Stream on the release population. In November we ran a Shield study to compare engagement of different aspects of the existing <a href="about:newtab" class="">about:newtab</a> with Activity Stream. The results are here:</div><div class=""><br class=""></div><div class=""><a href="https://sql.telemetry.mozilla.org/dashboard/activity-stream-a-b-testing-shield-study" class="">https://sql.telemetry.mozilla.org/dashboard/activity-stream-a-b-testing-shield-study</a></div><div class=""><br class=""></div><div class="">The take-away from that study was that the Activity Stream interface was lagging the existing interface in one key area: Activity Stream Top Sites vs. ‘frecent’ History Tiles. Our work since then has been to improve the positive engagement metric of Activity Stream. We have developed new Top Site selection algorithms and UIs and are running additional Test Pilot A/B tests to optimize the metric. We will be starting a second Shield study this month to compare it against the Tiles newtab on a production audience.</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 13, 2017, at 10:41 PM, Justin Dolske <<a href="mailto:dolske@mozilla.com" class="">dolske@mozilla.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Maybe I misunderstood. I was asking if you had a summary of findings relative to Firefox as it ships today. That is, when Activity Stream ships in Release, what kinds of improvements in user retention/engagement are we expecting to see from that?<br class=""><div class=""><div class=""><div class="gmail_extra"><br class=""></div><div class="gmail_extra">Justin<br class=""><br class=""></div><div class="gmail_extra"><div class="gmail_quote">On Mon, Feb 13, 2017 at 6:10 PM, Tim Spurway <span dir="ltr" class=""><<a href="mailto:tspurway@mozilla.com" target="_blank" class="">tspurway@mozilla.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">Hey Justin,<div class=""><br class=""></div><div class="">We have done a bunch of experiments (10 to date) that A/B test single features we are trying to introduce. Here is an example of one of them that we are running right now:</div><div class=""><br class=""></div><div class=""><a href="https://sql.telemetry.mozilla.org/dashboard/activity-stream-a-b-testing-deduped-combined-frecency" target="_blank" class="">https://sql.telemetry.mozilla.<wbr class="">org/dashboard/activity-stream-<wbr class="">a-b-testing-deduped-combined-<wbr class="">frecency</a></div><div class=""><br class=""></div><div class="">We are trying to figure out how to get better engagement on our “top sites” (frecent) tiles. We tried a tweak to the algorithm to better ‘de-dupe’ the top site results. The experiment is showing better positive engagement, so we have decided to run with the new ‘de-duped’ algorithm. <br class=""><div class=""><br class=""></div><div class="">If you look at the 'Dashboards > Activity Stream A/B Testing' menu in re:dash, you can see all of the dashboards for our experiments. I am the first to admit that our experiment documentation is a bit lacking, and it’s something we are working on improving this quarter as our results are being analyzed outside of our immediate product dev circle.</div><div class=""><div class="h5"><div class=""><br class=""></div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Feb 13, 2017, at 6:08 PM, Justin Dolske <<a href="mailto:dolske@mozilla.com" target="_blank" class="">dolske@mozilla.com</a>> wrote:</div><br class="m_3092161644506299170Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Feb 13, 2017 at 6:57 AM, Tim Spurway <span dir="ltr" class=""><<a href="mailto:tspurway@mozilla.com" target="_blank" class="">tspurway@mozilla.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">[...]<br class="">
Activity Stream has been developed using a series of A/B test based experiments, both on an opt-in Test Pilot and an opt-out production Shield audience. Our methodology is to continue optimizing the user experience along the user engagement and user retention dimensions. The <a class="">about:newtab</a> and <a class="">about:home</a> pages that land in Firefox will be measurably more engaging than the existing experience.<br class="">
<br class=""></blockquote><div class=""><br class=""></div><div class="">Because I'm curious but haven't been paying attention: is there anything that summarizes the findings so far? (e.g. an X% increase in new user retention, Y% increase in Firefox usage hours, or whatever?)<br class=""><br class=""></div><div class="">Justin <br class=""></div></div><br class=""></div></div>
</div></blockquote></div><br class=""></div></div></div></div></div></blockquote></div><br class=""></div></div></div></div>
</div></blockquote></div><br class=""></div></body></html>