Minimum Viable Metrics

Adam Rogers arogers at mozilla.com
Wed Nov 20 05:27:34 PST 2013


Agreed.  

Even half the MVM as you propose would provide great insights (and will be WAY more data that I'm used to getting :) )



----- Original Message -----
From: "Lloyd Hilaiel" <lloyd at mozilla.com>
To: "Katie Parlante" <kparlante at mozilla.com>, dev-fxacct at mozilla.org
Cc: "Adam Rogers" <arogers at mozilla.com>
Sent: Wednesday, November 20, 2013 5:25:28 AM
Subject: Re: Minimum Viable Metrics

I love the definition of mwm.

Generally speaking, I would sacrifice features and insights in order to minimize risk of having these basic numbers available and testable sooner...  Then iterate!

Lloyd

Katie Parlante <kparlante at mozilla.com> wrote:
>Hi all,
>
>I'm working on the FxAcct metrics, and I wanted to sketch out a high
>level plan for "Minimum Viable Metrics" -- the metrics required by the
>time the first accounts go live.
>
>The assumption here is that the first accounts will be for Marketplace
>and WMF, on FxOS and Web. That may change given the dates for FxOS 1.4,
>but its a reasonable place to start.
>
>The  current proposal for MVM:
>- % completion rate for FTU sign up flow
>- % completion rate for FxA creation (regardless of origin)
>- basic FxA event metrics (e.g. new accounts per day, email
>verifications per day, etc.)
>- basic fraud detection related data collection
>- self service access to all high level "metrics" data from Kibana (or
>similar, if need be)
>- Usable "Dashboard" for the key metrics (either customized in Kibana
>or some other solution)
>
>I suspect these are also going to be important/necessary before going
>live:
>- Performance data collected and accessible in "self serve" mode
>- Fraud detection data routed somewhere sensible (e.g. CEF logging
>routed to Arcsight)
>
>While we might not strictly require these next items for MVM, I think
>we can likely get much of this in (or at least the data logged) and we
>should design for them from the beginning: 
>- % completion rate for all flows related to initial services/devices
>- drop off rates for all flows related to initial services/devices
>- segmentation of flows by: device (FxOS, Android, Desktop, Web, etc.),
>service where flow originated (FTU, Marketplace, FxOS Settings, etc.),
>locale
>- "meta" flows that may span devices (e.g. create account --> email
>verification --> successful use of a service at some later point in
>time)
>
>Probably not in MVM
>- data routed from the Persona verifier. While this may count as
>"basic" data for the service, I suspect the overhead is not going to be
>worth it for the payoff in MVM. We can debate this, though.
>
>Constraints/Assumptions for MVM
>- No telemetry from the device/client, no separate API call just for
>metrics
>- No collaboration with other metrics efforts (like Marketplace)
>- Metrics may request additional info (optional?) be sent with existing
>API calls to get the segmentations above (though we may be able to get
>this data without it)
>- Use existing logging infrastructure (Heka + ElasticSearch + Kibana)
>
>Note that I'm not really covering all logging or monitoring here. (If
>there are other logging/monitoring/data projects where you'd like my
>help scoping out and formalizing requirements, let me know).
>
>Initial Milestones for MVM: 
>
>0. Proposal for events logged for FTU & FxA creation flows, and
>proposal for list of basic metrics (separate email & gh issues -- this
>week)
>
>1. End to End Proof of concept -- self service access to some basic
>metrics + FTU flow data. (December?)
>To answer existing questions (do we need to write a custom dashboard
>for UX and PM's looking at key metrics?) and flush out new ones
>(unknown unknowns!), I think we need to get the whole metrics universe
>up and running end-to-end. December may be too aggressive -- it depends
>on a staging stack with Heka + ElasticSearch + Kibana being up and
>running, for starters. The point is, I think we should race through a
>first pass and then figure out what needs filling in.
>
>Sound sane?
>
>The info above + more milestones & roadmap here:
>https://id.etherpad.mozilla.org/fxacct-mvm.
>
>Cheers,
>Katie
>
>
> 
>
>
> 
>
>
>_______________________________________________
>Dev-fxacct mailing list
>Dev-fxacct at mozilla.org
>https://mail.mozilla.org/listinfo/dev-fxacct

lloyd - (on an tiny keyboard)



More information about the Dev-fxacct mailing list