Minimum Viable Metrics
kparlante at mozilla.com
Wed Nov 20 12:13:00 PST 2013
Thanks for the specifics about what QA/SRE needs, thats very helpful.
On Nov 20, 2013, at 10:33 AM, Edwin Wong <edwong at mozilla.com> wrote:
> A twist on the % completion/failure rates is the discreet number of users with long load or query times. This would add give us a better view of site performance and if we are meeting our UX goals. I don't want to lose valuable data in the 1%.
> I also would push for having specific questions we want the data to answer. For QA I'm thinking:
> As a QA/SRE I need the exact number of failed (error) sign in per day so that I can be confident that our deployment is successful.
I think you also need to know the number of attempts to put the number of errors in context. I hear you that exact numbers are important for QA/SRE, not just rates. (jrgm has mentioned this as well).
> As a QA/SRE I need the long load times (over threshold) by region so that we know our performance is providing a good UX.
> As a QA/SRE I need the number of user time outs by region so that we know our performance is providing a good UX.
Taras suggested giving perf numbers as 50th percentile and 95th percentile. We could also show a histogram so that you could see the distribution (including what's over threshold).
I'll add segmentation by region as a requirement for perf data. (Do we want this segmentation for everything?)
"User time outs" wasn't on my radar yet, so thanks for bringing that up.
Agreed that specific questions are important. The completion rates are meant to focus the whole group on the overarching question/problem: are users successful? (At FTU, or creating a FxA, etc.) The other stats (segmentations, flow drop off rates, perf and error numbers) help us dig into it more deeply.
More information about the Dev-fxacct