Minimum Viable Metrics
kparlante at mozilla.com
Tue Nov 19 16:20:34 PST 2013
I'm working on the FxAcct metrics, and I wanted to sketch out a high level plan for "Minimum Viable Metrics" -- the metrics required by the time the first accounts go live.
The assumption here is that the first accounts will be for Marketplace and WMF, on FxOS and Web. That may change given the dates for FxOS 1.4, but its a reasonable place to start.
The current proposal for MVM:
- % completion rate for FTU sign up flow
- % completion rate for FxA creation (regardless of origin)
- basic FxA event metrics (e.g. new accounts per day, email verifications per day, etc.)
- basic fraud detection related data collection
- self service access to all high level "metrics" data from Kibana (or similar, if need be)
- Usable "Dashboard" for the key metrics (either customized in Kibana or some other solution)
I suspect these are also going to be important/necessary before going live:
- Performance data collected and accessible in "self serve" mode
- Fraud detection data routed somewhere sensible (e.g. CEF logging routed to Arcsight)
While we might not strictly require these next items for MVM, I think we can likely get much of this in (or at least the data logged) and we should design for them from the beginning:
- % completion rate for all flows related to initial services/devices
- drop off rates for all flows related to initial services/devices
- segmentation of flows by: device (FxOS, Android, Desktop, Web, etc.), service where flow originated (FTU, Marketplace, FxOS Settings, etc.), locale
- "meta" flows that may span devices (e.g. create account --> email verification --> successful use of a service at some later point in time)
Probably not in MVM
- data routed from the Persona verifier. While this may count as "basic" data for the service, I suspect the overhead is not going to be worth it for the payoff in MVM. We can debate this, though.
Constraints/Assumptions for MVM
- No telemetry from the device/client, no separate API call just for metrics
- No collaboration with other metrics efforts (like Marketplace)
- Metrics may request additional info (optional?) be sent with existing API calls to get the segmentations above (though we may be able to get this data without it)
- Use existing logging infrastructure (Heka + ElasticSearch + Kibana)
Note that I'm not really covering all logging or monitoring here. (If there are other logging/monitoring/data projects where you'd like my help scoping out and formalizing requirements, let me know).
Initial Milestones for MVM:
0. Proposal for events logged for FTU & FxA creation flows, and proposal for list of basic metrics (separate email & gh issues -- this week)
1. End to End Proof of concept -- self service access to some basic metrics + FTU flow data. (December?)
To answer existing questions (do we need to write a custom dashboard for UX and PM's looking at key metrics?) and flush out new ones (unknown unknowns!), I think we need to get the whole metrics universe up and running end-to-end. December may be too aggressive -- it depends on a staging stack with Heka + ElasticSearch + Kibana being up and running, for starters. The point is, I think we should race through a first pass and then figure out what needs filling in.
The info above + more milestones & roadmap here: https://id.etherpad.mozilla.org/fxacct-mvm.
More information about the Dev-fxacct