<html><head></head><body>I love the definition of mwm.<br>
<br>
Generally speaking, I would sacrifice features and insights in order to minimize risk of having these basic numbers available and testable sooner... Then iterate!<br>
<br>
Lloyd<br><br><div class="gmail_quote">Katie Parlante <kparlante@mozilla.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="k9mail">Hi all,<br /><br />I'm working on the FxAcct metrics, and I wanted to sketch out a high level plan for "Minimum Viable Metrics" -- the metrics required by the time the first accounts go live.<br /><br />The assumption here is that the first accounts will be for Marketplace and WMF, on FxOS and Web. That may change given the dates for FxOS 1.4, but its a reasonable place to start.<br /><br />The current proposal for MVM:<br />- % completion rate for FTU sign up flow<br />- % completion rate for FxA creation (regardless of origin)<br />- basic FxA event metrics (e.g. new accounts per day, email verifications per day, etc.)<br />- basic fraud detection related data collection<br />- self service access to all high level "metrics" data from Kibana (or similar, if need be)<br />- Usable "Dashboard" for the key metrics (either customized in Kibana or some other solution)<br /><br />I suspect these are also going to be important/necessary before going live:<br />-
Performance data collected and accessible in "self serve" mode<br />- Fraud detection data routed somewhere sensible (e.g. CEF logging routed to Arcsight)<br /><br />While we might not strictly require these next items for MVM, I think we can likely get much of this in (or at least the data logged) and we should design for them from the beginning: <br />- % completion rate for all flows related to initial services/devices<br />- drop off rates for all flows related to initial services/devices<br />- segmentation of flows by: device (FxOS, Android, Desktop, Web, etc.), service where flow originated (FTU, Marketplace, FxOS Settings, etc.), locale<br />- "meta" flows that may span devices (e.g. create account --> email verification --> successful use of a service at some later point in time)<br /><br />Probably not in MVM<br />- data routed from the Persona verifier. While this may count as "basic" data for the service, I suspect the overhead is not going to be worth it for the
payoff in MVM. We can debate this, though.<br /><br />Constraints/Assumptions for MVM<br />- No telemetry from the device/client, no separate API call just for metrics<br />- No collaboration with other metrics efforts (like Marketplace)<br />- Metrics may request additional info (optional?) be sent with existing API calls to get the segmentations above (though we may be able to get this data without it)<br />- Use existing logging infrastructure (Heka + ElasticSearch + Kibana)<br /><br />Note that I'm not really covering all logging or monitoring here. (If there are other logging/monitoring/data projects where you'd like my help scoping out and formalizing requirements, let me know).<br /><br />Initial Milestones for MVM: <br /><br />0. Proposal for events logged for FTU & FxA creation flows, and proposal for list of basic metrics (separate email & gh issues -- this week)<br /><br />1. End to End Proof of concept -- self service access to some basic metrics + FTU flow data.
(December?)<br />To answer existing questions (do we need to write a custom dashboard for UX and PM's looking at key metrics?) and flush out new ones (unknown unknowns!), I think we need to get the whole metrics universe up and running end-to-end. December may be too aggressive -- it depends on a staging stack with Heka + ElasticSearch + Kibana being up and running, for starters. The point is, I think we should race through a first pass and then figure out what needs filling in.<br /><br />Sound sane?<br /><br />The info above + more milestones & roadmap here: <a href="https://id.etherpad.mozilla.org/fxacct-mvm">https://id.etherpad.mozilla.org/fxacct-mvm</a>.<br /><br />Cheers,<br />Katie<br /><br /><br /><br /><br /><br /><br /><br /><br /><hr /><br />Dev-fxacct mailing list<br />Dev-fxacct@mozilla.org<br /><a href="https://mail.mozilla.org/listinfo/dev-fxacct">https://mail.mozilla.org/listinfo/dev-fxacct</a><br /></pre></blockquote></div><br>
lloyd - (on an tiny keyboard)</body></html>