<div dir="ltr">Thanks all, for your analysis.<div><br></div><div>After reading the thread, I support the idea that we just log the uid verbatim and let the metrics team do the appropriate scrubbing/obfuscation, if needed. It’s the simplest approach, and going forward, I see many headaches with making sure the uid is derived the same way across all the services and clients. </div><div><br></div><div>I probably proposed the original notion of doing the metrics on the “edge”. It might be a good idea, but for now, it seems to create more complication than it’s worth.</div><div><br></div><div>In term of privacy, IMO, we need to be most concerned with long term retention of data that creates risk if it’s exposed. I assume all this log data (and derived data) is stored for a limited time period. </div><div><br></div><div>-chris</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 13, 2015 at 3:51 PM, Katie Parlante <span dir="ltr"><<a href="mailto:kparlante@mozilla.com" target="_blank">kparlante@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Mostly I'm just agreeing with Ryan below...<span class=""><br>
<br>
On 5/11/15 9:32 PM, Ryan Kelly wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 12/05/2015 13:14, Andrew Chilton wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 5 May 2015 at 17:33, Ryan Kelly <<a href="mailto:rfkelly@mozilla.com" target="_blank">rfkelly@mozilla.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
One of the tricky-but-important questions we need to answer is:<br>
<br>
* how many users accessed more than one FxA service this month?<br>
</blockquote>
<br>
Like *any* two, or a specific two e.g. Sync and Hello? (Or three I suppose.)<br>
</blockquote>
<br>
I was thinking "accessed any two services". Bu correlating between<br>
specific services also sounds useful.<br>
</blockquote>
<br></span>
FWIW, the primary use case that the executives care about is "any two services". One can image that someone may eventually ask about a specific two.<div><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Can we do it in a more privacy-conscious manner?<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Perhaps we could<br>
post-process these in the data pipeline into something else, or we can<br>
log something locally which we could use to correlate that same user to<br>
another service (but not back to the user him/herself). The idea of a<br>
Metrics ID has been raised which is a one-way mapping from uid to<br>
Metrics ID (am leaving out any implementation details for now).<br>
</blockquote></blockquote>
[...snip...]<br>
<br>
From what we are looking at above (i.e. "How many ...?") questions,<br>
then is it safe to assume we won't be asked for answers to questions<br>
such as "Who has ...?". i.e. are we always going to respond with an<br>
aggregated number such as 6,000,001 rather than a list of users? If we<br>
do the Metrics ID then we can't answer the "Who has ...?" questions<br>
anyway, so are we sure we won't need to provide these kinds of<br>
answers? And if we are asked to provide such answers, should we even<br>
allow that (based on protecting the users privacy)?<br>
</blockquote>
<br>
I don't think we need to answer "who did X?" questions on a post-hoc<br>
basis, and in fact we should actively try to be unable to answer such<br>
questions.<br>
<br>
We may want to know "user XYZ just did X" on a real-time basis under<br>
very controlled circumstances, e.g. to trigger product<br>
marketing/engagement emails for users who have opted into them. But I<br>
hope we can deal with such events as they occur and then forget about<br>
them, rather than building up a big database of individual user activity<br>
over time.<br>
<br>
</blockquote>
<br></div></div>
Agree with Ryan. The account activity events are meant to answer "How Many" questions.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Of course, all services would need to know how to make that MetricsID if it<br>
was logged at the edge, but if the uid was post-processed in the data<br>
pipeline this could be done centrally.<br>
</blockquote>
<br>
Yep. If every service is able to do the uid -> metrics-id mapping at<br>
will, then does it really gain us anything?<br>
</blockquote>
<br>
Not really. I'm definitely a +1 on doing the metrics-id in post<br>
processing so that each edge can just log uid as-is. I believe Heka<br>
currently scrubs UIDs and emails from the fxa-auth-server logs so<br>
converting to a metrics id and scrubbing the original uid seems<br>
possible.<br>
</blockquote>
<br>
I'm starting to like this approach as well, as it seems to simplify<br>
things while still taking the anonymization issue seriously.<br>
<br>
It would also make the aforementioned engagement integration easier.<br>
</blockquote>
<br></span>
+1<br>
<br>
As I understand it, the primary goal of the metrics-id is that the group of people who are working with the metrics data set don't have access to the uid or email, which limits risk.<br>
<br>
Could we go ahead and use metrics-id on all log data where we currently scrub account uids? That would unlock some potentially valuable longitudinal analysis & debugging. Alternatively, a session-id would be almost as useful.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'd love for people to weigh in with their gut reactions here, even if<br>
you don't have any comments on the technical details.<br>
<br>
We will of course have to be in compliance with Mozilla's terms, privacy<br>
policy, etc when collecting all these metrics. But IMHO saying "we're<br>
compliant with the posted ToS!" is not much help if what we're doing<br>
just feels wrong to people.<br>
</blockquote>
<br>
I think you're right about the 'if it just feels wrong' however, how<br>
do we actually go about measuring it against the manifesto (et al)? Is<br>
it just our gut feel which tells us if we're doing fine against it?<br>
</blockquote>
<br>
The combination of "in compliance with our posted legal policies" and<br>
"feels about right" is IMHO a very good start. We can run it by our<br>
legal/policy team once we've got a concrete proposal.<br>
</blockquote>
<br></span>
Seems like "metrics-id" would fall in a similar category to Hello's roomid: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1140184" target="_blank">https://bugzilla.mozilla.org/show_bug.cgi?id=1140184</a>, and the way we treat log data in general. Even though we take some steps to sanitize it, we do not treat log data as perfectly safe, we still have controls around who can access it and we monitor access.<br>
<br>
Cheers,<br>
Katie<div class="HOEnZb"><div class="h5"><br>
<br>
_______________________________________________<br>
Dev-fxacct mailing list<br>
<a href="mailto:Dev-fxacct@mozilla.org" target="_blank">Dev-fxacct@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/dev-fxacct" target="_blank">https://mail.mozilla.org/listinfo/dev-fxacct</a><br>
</div></div></blockquote></div><br></div>