Minimum Viable Metrics

Edwin Wong edwong at
Wed Nov 20 10:33:19 PST 2013

Hi Katie,

A twist on the % completion/failure rates is the discreet number of users with long load or query times.  This would add give us a better view of site performance and if we are meeting our UX goals. I don't want to lose valuable data in the 1%.  

I also would push for having specific questions we want the data to answer.  For QA I'm thinking:
As a QA/SRE I need the exact number of failed (error) sign in per day so that I can be confident that our deployment is successful.
As a QA/SRE I need the long load times (over threshold) by region so that we know our performance is providing a good UX.
As a QA/SRE I need the number of user time outs by region so that we know our performance is providing a good UX.


----- Original Message -----
From: "Katie Parlante" <kparlante at>
To: dev-fxacct at
Cc: "Adam Rogers" <arogers at>
Sent: Tuesday, November 19, 2013 4:20:34 PM
Subject: Minimum Viable Metrics

Hi all,

I'm working on the FxAcct metrics, and I wanted to sketch out a high level plan for "Minimum Viable Metrics" -- the metrics required by the time the first accounts go live.

The assumption here is that the first accounts will be for Marketplace and WMF, on FxOS and Web. That may change given the dates for FxOS 1.4, but its a reasonable place to start.

The  current proposal for MVM:
- % completion rate for FTU sign up flow
- % completion rate for FxA creation (regardless of origin)
- basic FxA event metrics (e.g. new accounts per day, email verifications per day, etc.)
- basic fraud detection related data collection
- self service access to all high level "metrics" data from Kibana (or similar, if need be)
- Usable "Dashboard" for the key metrics (either customized in Kibana or some other solution)

I suspect these are also going to be important/necessary before going live:
- Performance data collected and accessible in "self serve" mode
- Fraud detection data routed somewhere sensible (e.g. CEF logging routed to Arcsight)

While we might not strictly require these next items for MVM, I think we can likely get much of this in (or at least the data logged) and we should design for them from the beginning: 
- % completion rate for all flows related to initial services/devices
- drop off rates for all flows related to initial services/devices
- segmentation of flows by: device (FxOS, Android, Desktop, Web, etc.), service where flow originated (FTU, Marketplace, FxOS Settings, etc.), locale
- "meta" flows that may span devices (e.g. create account --> email verification --> successful use of a service at some later point in time)

Probably not in MVM
- data routed from the Persona verifier. While this may count as "basic" data for the service, I suspect the overhead is not going to be worth it for the payoff in MVM. We can debate this, though.

Constraints/Assumptions for MVM
- No telemetry from the device/client, no separate API call just for metrics
- No collaboration with other metrics efforts (like Marketplace)
- Metrics may request additional info (optional?) be sent with existing API calls to get the segmentations above (though we may be able to get this data without it)
- Use existing logging infrastructure (Heka + ElasticSearch + Kibana)

Note that I'm not really covering all logging or monitoring here. (If there are other logging/monitoring/data projects where you'd like my help scoping out and formalizing requirements, let me know).

Initial Milestones for MVM: 

0. Proposal for events logged for FTU & FxA creation flows, and proposal for list of basic metrics (separate email & gh issues -- this week)

1. End to End Proof of concept -- self service access to some basic metrics + FTU flow data. (December?)
To answer existing questions (do we need to write a custom dashboard for UX and PM's looking at key metrics?) and flush out new ones (unknown unknowns!), I think we need to get the whole metrics universe up and running end-to-end. December may be too aggressive -- it depends on a staging stack with Heka + ElasticSearch + Kibana being up and running, for starters. The point is, I think we should race through a first pass and then figure out what needs filling in.

Sound sane?

The info above + more milestones & roadmap here:




Dev-fxacct mailing list
Dev-fxacct at

More information about the Dev-fxacct mailing list