A/B Testing in Firefox for Android

Mark Finkle mfinkle at mozilla.com
Thu Sep 3 18:16:42 UTC 2015


We have decided to start running A/B Testing [1] in Firefox for Android.
These experiments are intended to optimize specific outcomes, as well as,
inform our long-term design decisions. We want to create the best Firefox
experience we can, and these experiments will help.

The system will also allow us to throttle the release of features, called
staged rollout, so we can monitor new features in a controlled manner
across a large userbase and a fragmented device ecosystem. If we need to
rollback a feature for some reason, we'd have the ability to do that.

Technical details:
* Switchboard is used to control experiment segmenting and staged rollout.
* Telemetry is used to collect metrics about an experiment.
* FHR is used to track active experiments so we can correlate to
application usage.

== What is Switchboard? ==

Switchboard is an open source SDK for doing A/B testing and staged rollouts
[2][3]. It connects to a server component, which maintains a list of active
experiments.

The SDK does create a UUID, which is stored on the device. The UUID is sent
to the server, which uses it to "bucket" the client, but the UUID is never
stored on the server. In fact, the server does not store any data. The
server we are using is being hosted by Mozilla.

We decided to start using Switchboard because it's simple, open source,
saves no data and can be hosted by Mozilla. Thanks to the KeepSafe folks
for releasing Switchboard.

== Planning Experiments ==

The Mobile Product and UX teams are the primary drivers for creating
experiments, but as is common on the Mobile team, ideas can come from
anywhere. We have been working with the Mozilla Growth team, getting a
better understanding of how to design the experiments and analyze the
metrics. UX researchers will also have input into the experiments.

Once Product and UX complete the experiment design, Development would land
code in Firefox to implement the desired variations of the experiment.
Development would also land code in the Switchboard server to control the
configuration of the experiment: Is it active? How are the variations
distributed across the user population?

Since we use Telemetry to collect metrics on the experiments, the Beta
channel is likely our best time period to run experiments. Telemetry is on
by default on Nightly, Aurora and Beta; and Beta is the largest userbase of
those three channels.

Once we decide which variation of the experiment is the "winner", we'll
change the Switchboard server configuration for the experiment so that 100%
of the userbase will flow through the winning variation.

Yes, a small percentage of the Release channel has Telemetry enabled, but
it might be too small to be useful for experimentation. Time will tell.

Note: Switchboard itself will be enabled on all channels. It collects no
data and gives us a "code-free" way of staging rollouts. It much less risky
and time consuming than uplifting patches that need to land on branches at
specific times.

== What's Happening Now? ==

Switchboard has already landed in Nightly [4] and is currently behind a
Nightly build flag. Once we feel comfortable, we'll let it ride the trains.
Our first experiment will likely be testing a new onboarding experience [5].

[1] https://en.wikipedia.org/wiki/A/B_testing
[2] https://github.com/KeepSafe/Switchboard
[3]
http://keepsafe-engineering.tumblr.com/post/28437940369/easy-mobile-ab-testing
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1196897
[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1199859
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/mobile-firefox-dev/attachments/20150903/2c784c85/attachment.html>


More information about the mobile-firefox-dev mailing list