Minimum Viable Metrics

Lloyd Hilaiel lloyd at
Wed Nov 20 02:25:28 PST 2013

I love the definition of mwm.

Generally speaking, I would sacrifice features and insights in order to minimize risk of having these basic numbers available and testable sooner...  Then iterate!


Katie Parlante <kparlante at> wrote:
>Hi all,
>I'm working on the FxAcct metrics, and I wanted to sketch out a high
>level plan for "Minimum Viable Metrics" -- the metrics required by the
>time the first accounts go live.
>The assumption here is that the first accounts will be for Marketplace
>and WMF, on FxOS and Web. That may change given the dates for FxOS 1.4,
>but its a reasonable place to start.
>The  current proposal for MVM:
>- % completion rate for FTU sign up flow
>- % completion rate for FxA creation (regardless of origin)
>- basic FxA event metrics (e.g. new accounts per day, email
>verifications per day, etc.)
>- basic fraud detection related data collection
>- self service access to all high level "metrics" data from Kibana (or
>similar, if need be)
>- Usable "Dashboard" for the key metrics (either customized in Kibana
>or some other solution)
>I suspect these are also going to be important/necessary before going
>- Performance data collected and accessible in "self serve" mode
>- Fraud detection data routed somewhere sensible (e.g. CEF logging
>routed to Arcsight)
>While we might not strictly require these next items for MVM, I think
>we can likely get much of this in (or at least the data logged) and we
>should design for them from the beginning: 
>- % completion rate for all flows related to initial services/devices
>- drop off rates for all flows related to initial services/devices
>- segmentation of flows by: device (FxOS, Android, Desktop, Web, etc.),
>service where flow originated (FTU, Marketplace, FxOS Settings, etc.),
>- "meta" flows that may span devices (e.g. create account --> email
>verification --> successful use of a service at some later point in
>Probably not in MVM
>- data routed from the Persona verifier. While this may count as
>"basic" data for the service, I suspect the overhead is not going to be
>worth it for the payoff in MVM. We can debate this, though.
>Constraints/Assumptions for MVM
>- No telemetry from the device/client, no separate API call just for
>- No collaboration with other metrics efforts (like Marketplace)
>- Metrics may request additional info (optional?) be sent with existing
>API calls to get the segmentations above (though we may be able to get
>this data without it)
>- Use existing logging infrastructure (Heka + ElasticSearch + Kibana)
>Note that I'm not really covering all logging or monitoring here. (If
>there are other logging/monitoring/data projects where you'd like my
>help scoping out and formalizing requirements, let me know).
>Initial Milestones for MVM: 
>0. Proposal for events logged for FTU & FxA creation flows, and
>proposal for list of basic metrics (separate email & gh issues -- this
>1. End to End Proof of concept -- self service access to some basic
>metrics + FTU flow data. (December?)
>To answer existing questions (do we need to write a custom dashboard
>for UX and PM's looking at key metrics?) and flush out new ones
>(unknown unknowns!), I think we need to get the whole metrics universe
>up and running end-to-end. December may be too aggressive -- it depends
>on a staging stack with Heka + ElasticSearch + Kibana being up and
>running, for starters. The point is, I think we should race through a
>first pass and then figure out what needs filling in.
>Sound sane?
>The info above + more milestones & roadmap here:
>Dev-fxacct mailing list
>Dev-fxacct at

lloyd - (on an tiny keyboard)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the Dev-fxacct mailing list