<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hey Justin,<div class=""><br class=""></div><div class="">We have done a bunch of experiments (10 to date) that A/B test single features we are trying to introduce. Here is an example of one of them that we are running right now:</div><div class=""><br class=""></div><div class=""><a href="https://sql.telemetry.mozilla.org/dashboard/activity-stream-a-b-testing-deduped-combined-frecency" class="">https://sql.telemetry.mozilla.org/dashboard/activity-stream-a-b-testing-deduped-combined-frecency</a></div><div class=""><br class=""></div><div class="">We are trying to figure out how to get better engagement on our “top sites” (frecent) tiles. We tried a tweak to the algorithm to better ‘de-dupe’ the top site results. The experiment is showing better positive engagement, so we have decided to run with the new ‘de-duped’ algorithm. <br class=""><div class=""><br class=""></div><div class="">If you look at the 'Dashboards > Activity Stream A/B Testing' menu in re:dash, you can see all of the dashboards for our experiments. I am the first to admit that our experiment documentation is a bit lacking, and it’s something we are working on improving this quarter as our results are being analyzed outside of our immediate product dev circle.</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 13, 2017, at 6:08 PM, Justin Dolske <<a href="mailto:dolske@mozilla.com" class="">dolske@mozilla.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Feb 13, 2017 at 6:57 AM, Tim Spurway <span dir="ltr" class=""><<a href="mailto:tspurway@mozilla.com" target="_blank" class="">tspurway@mozilla.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">[...]<br class="">
Activity Stream has been developed using a series of A/B test based experiments, both on an opt-in Test Pilot and an opt-out production Shield audience. Our methodology is to continue optimizing the user experience along the user engagement and user retention dimensions. The <a href="about:newtab" class="">about:newtab</a> and <a href="about:home" class="">about:home</a> pages that land in Firefox will be measurably more engaging than the existing experience.<br class="">
<br class=""></blockquote><div class=""><br class=""></div><div class="">Because I'm curious but haven't been paying attention: is there anything that summarizes the findings so far? (e.g. an X% increase in new user retention, Y% increase in Firefox usage hours, or whatever?)<br class=""><br class=""></div><div class="">Justin <br class=""></div></div><br class=""></div></div>
</div></blockquote></div><br class=""></div></div></body></html>