<div dir="ltr">> So how can we make the gathering of these metrics feel as<br>> privacy-sensitive, as safe, as *right* as possible?<br><br>I'm not going to comment on what feels "creepy" or "right", only that I was<br>reminded of the following as I read through Ryan's concerns.<br><br>A while back Mayo pointed us at RAPPOR[1][2], a "novel privacy technology that<br>allows inferring statistics about populations while preserving the privacy of<br>individual users". The approach RAPPOR takes works better with data submitted<br>by the client (e.g. our telemetry system) than data derived from server logs,<br>as any particular application server will only be able to log events for users<br>that actually use the service.<br><br>-whd<br><br>[1] <a href="https://github.com/google/rappor">https://github.com/google/rappor</a><br>[2] <a href="http://googleresearch.blogspot.com/2014/10/learning-statistics-with-privacy-aided.html">http://googleresearch.blogspot.com/2014/10/learning-statistics-with-privacy-aided.html</a><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 4, 2015 at 10:33 PM, Ryan Kelly <span dir="ltr"><<a href="mailto:rfkelly@mozilla.com" target="_blank">rfkelly@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 4/05/2015 15:23, Andrew Chilton wrote:<br>
> I'm currently looking at FxA user metrics this quarter and have a few<br>
> thoughts around some ideas. Let me kick off by just having a brief<br>
> outline so you get an idea of what's needed.<br>
><br>
> "All services attached to FxA should report metrics on user<br>
> activity in a consistent, privacy-respecting manner. We should have<br>
> dashboards that allow us to measure success and monitor for problems."<br>
<br>
</span>Thanks for kicking this off Andy, getting better metrics is a big part<br>
of our vision for a successful Q2. You and I have chatted a bit about<br>
this IRL, but I'll add some more notes and context below for the benefit<br>
of discussion on the list.<br>
<span class=""><br>
> This may or may not be the exact goal we're reaching for, but it's<br>
> currently a good statement to aim for. Other questions we might like to<br>
> ask and answer in the future are:<br>
><br>
> * how many users signed up this month?<br>
> * how many users used a particular service (e.g. Hello, Sync)?<br>
> * how many users logged in with a mobile device?<br>
> * (perhaps other questions we don't yet know about)<br>
<br>
</span>One of the tricky-but-important questions we need to answer is:<br>
<br>
* how many users accessed more than one FxA service this month?<br>
<br>
It's worth calling this one out explicitly, because this is why we need<br>
to somehow correlate user activity across services.<br>
<span class=""><br>
> To do this we need to figure out how to make this happen. By logging<br>
> 'user events' we should be able to take advantage of the regular data<br>
> pipeline running through Heka/ElasticSearch/Kibana and/or any other<br>
> Reporting/MapReduce plus custom dashoards as aimed for by the Data<br>
> Pipeline v2 [1]. Therefore this email is mainly concentrating on what we<br>
> do at the edges, i.e. our application servers.<br>
><br>
> We would love to correlate users across services so we need something<br>
> which allows us to do this. Of course the uid of a user is the obvious<br>
> answer but one that would raise some privacy questions.<br>
<br>
</span>Right, so the simplest thing would be for each service to just emit a<br>
bunch of JSON log entries like this:<br>
<br>
<a href="http://log.info" target="_blank">log.info</a>({<br>
service: "hello"<br>
uid: "ABCDEF123456"<br>
event: "call",<br>
timestamp: 1430802399476,<br>
})<br>
<br>
<a href="http://log.info" target="_blank">log.info</a>({<br>
service: "readinglist"<br>
uid: "ABCDEF123456"<br>
event: "save_item",<br>
timestamp: 1430803546071,<br>
})<br>
<br>
Heka could slurp this up and send them off for processing/aggregation in<br>
the same way that we currently do for the existing FxA<br>
monthly-active-users count.<br>
<br>
Would it be OK for us to just go ahead and ship it this way?<br>
<br>
To me it seems a little creepy. If these events are being stored<br>
somewhere, you could potentially build up a pretty nice picture of an<br>
individual user's activity by analyzing their stream of events.<br>
Accidentally leaking metrics in this form would be a pretty big deal.<br>
<br>
Can we do it in a more privacy-conscious manner?<br>
<span class=""><br>
> Perhaps we could<br>
> post-process these in the data pipeline into something else, or we can<br>
> log something locally which we could use to correlate that same user to<br>
> another service (but not back to the user him/herself). The idea of a<br>
> Metrics ID has been raised which is a one-way mapping from uid to<br>
> Metrics ID (am leaving out any implementation details for now).<br>
<br>
</span>We have a tiny bit of prior art here, in the monthly-active-users<br>
counting for sync:<br>
<br>
<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1136014" target="_blank">https://bugzilla.mozilla.org/show_bug.cgi?id=1136014</a><br>
<br>
For this, we wound up emitting metrics events that look like:<br>
<br>
<a href="http://log.info" target="_blank">log.info</a>({<br>
uid: HMAC_SHA256(<secret key>, <uid>),<br>
timestamp: 1430802399476,<br>
...other sync-specific metrics...<br>
})<br>
<br>
In other words, we use HMAC to derive an opaque "metrics id" from the<br>
account uid. This lets us count unique users of the service, but makes<br>
it harder to correlate the logs with a particular user record from FxA.<br>
<br>
If all the services used the same technique, we could do cross-service<br>
activity correlation.<br>
<br>
I'd be interested in people's thoughts on the usefulness of this<br>
obfuscation.<br>
<span class=""><br>
> Of course, all services would need to know how to make that MetricsID if it<br>
> was logged at the edge, but if the uid was post-processed in the data<br>
> pipeline this could be done centrally.<br>
<br>
</span>Yep. If every service is able to do the uid -> metrics-id mapping at<br>
will, then does it really gain us anything?<br>
<br>
<br>
I'd love for people to weigh in with their gut reactions here, even if<br>
you don't have any comments on the technical details.<br>
<br>
We will of course have to be in compliance with Mozilla's terms, privacy<br>
policy, etc when collecting all these metrics. But IMHO saying "we're<br>
compliant with the posted ToS!" is not much help if what we're doing<br>
just feels wrong to people.<br>
<br>
So how can we make the gathering of these metrics feel as<br>
privacy-sensitive, as safe, as *right* as possible?<br>
<br>
<br>
Cheers,<br>
<br>
Ryan<br>
_______________________________________________<br>
Dev-fxacct mailing list<br>
<a href="mailto:Dev-fxacct@mozilla.org">Dev-fxacct@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/dev-fxacct" target="_blank">https://mail.mozilla.org/listinfo/dev-fxacct</a><br>
</blockquote></div><br></div>