<div dir="ltr">Hmm, clearly I'm missing bits here... Can you please point me somewhere to read more about the overall flow, and the UAC prompt and notificarions in particular? (Code pointers are fine too!)<br></div><div class="gmail_extra">
<br clear="all"><div>--<br>Ehsan<br><<a href="http://ehsanakhgari.org/">http://ehsanakhgari.org/</a>></div>
<br><br><div class="gmail_quote">On Tue, Jul 22, 2014 at 9:38 PM, Gregory Szorc <span dir="ltr"><<a href="mailto:gps@mozilla.com" target="_blank">gps@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The way the hotfix works is it downloaded the full installer and launched it (using a stub executable that requested UAC elevation, if necessary). There was no prompting inside Firefox itself. IMO it is a bit disturbing to see the UAC prompt come out of nowhere.<br>
<br>
We suspect that a large number of users wouldn't accept the UAC prompt on this first "random" display. We were banking on the notifications (clicking the button inside would trigger the UAC prompt) to get the high click through rate.<br>
<br>
I'm more concerned about the low percentage of notifications shown.<div class=""><br>
<br>
On 7/22/14, 6:29 PM, Ehsan Akhgari wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">
This looks great, thanks Greg!<br>
<br>
The "Elevation Cancelled" percentage frightens me. :( That makes me<br>
think we may never be able to get those users updated. Brian, Robert,<br>
is there any way we can get the update Windows service to work for these<br>
users?<br>
<br>
--<br>
Ehsan<br>
<<a href="http://ehsanakhgari.org/" target="_blank">http://ehsanakhgari.org/</a>><br>
<br>
<br>
On Tue, Jul 22, 2014 at 9:14 PM, Gregory Szorc <<a href="mailto:gps@mozilla.com" target="_blank">gps@mozilla.com</a><br></div><div><div class="h5">
<mailto:<a href="mailto:gps@mozilla.com" target="_blank">gps@mozilla.com</a>>> wrote:<br>
<br>
On 7/16/14, 11:27 AM, Gregory Szorc wrote:<br>
<br>
We just deployed a new Firefox hotfix to our entire user base.<br>
<br>
Hotfix v20140527.01 (tracked in bug 928173) identifies Firefox<br>
installs<br>
that are "stuck" on old releases and attempts to upgrade them to<br>
Firefox<br>
30 by downloading a full installer and running it, bypassing the<br>
built-in updating mechanism. The goal of this hotfix is to<br>
reduce the<br>
long tail of installs running older Firefox releases and get<br>
these users<br>
on a modern, safer, better, faster, stronger (but not harder)<br>
Firefox.<br>
<br>
The hotfix will be downloaded and installed on every Firefox 10+<br>
install. However, the upgrading only occurs if a number of<br>
conditions<br>
are met [1]. Essentially, Firefox < 29 on Windows x86<br>
non-partner builds<br>
on the release channel that have auto updating enabled. So it is<br>
stated<br>
explicitly, we only do the upgrade on clients that should have<br>
already<br>
upgraded automatically: we don't go behind our user's backs and<br>
force an<br>
unwanted update.<br>
<br>
If you see an uptick in new Firefox 30 users or an increase of<br>
surprised<br>
users that their Firefox is drastically different, this is<br>
likely the<br>
reason.<br>
<br>
If you see something strange, don't hesitate to pick up the<br>
phone and<br>
call Benjamin Smedberg or myself, especially if you think the hotfix<br>
should be pulled.<br>
<br>
[1]<br></div></div>
<a href="https://hg.mozilla.org/__releases/firefox-hotfixes/__file/d29a3d24405c/v20140527.__01/resource/update.jsm#l875" target="_blank">https://hg.mozilla.org/__<u></u>releases/firefox-hotfixes/__<u></u>file/d29a3d24405c/v20140527.__<u></u>01/resource/update.jsm#l875</a><div>
<div class="h5"><br>
<<a href="https://hg.mozilla.org/releases/firefox-hotfixes/file/d29a3d24405c/v20140527.01/resource/update.jsm#l875" target="_blank">https://hg.mozilla.org/<u></u>releases/firefox-hotfixes/<u></u>file/d29a3d24405c/v20140527.<u></u>01/resource/update.jsm#l875</a>><br>
<br>
<br>
I thought people would interested in learning how effective this<br>
hotfix has been and what was observed from the user base.<br>
<br>
Data Analyzed<br>
=============<br>
<br>
The update hotfix submits data to Mozilla upon uninstall and other<br>
key events.<br>
<br>
Data is only submitted if the hotfix was compatible to a particular<br>
install. e.g. if you were running Firefox 30 when the hotfix was<br>
installed, you didn't upload data.<br>
<br>
Data is only submitted if the user has consented to data upload.<br>
This effectively boils down to FHR or Telemetry being enabled. This<br>
means we have no direct hotfix data on some users. We account for<br>
users upgrading from pre-FHR Firefoxes and wait for FHR to kick in<br>
before we assume they can't submit data and uninstall. This allows<br>
us to maximize data return.<br>
<br>
Due to a bug in the hotfix (possibly Gecko) (bug 1040231) combined<br>
with a server that didn't react well to malformed data, *we lost<br>
about the first 24 hours of data*. This means we don't have direct<br>
data for hotfix installs that completed during this time. This is<br>
very unfortunate and adds a massive asterisk to our analysis.<br>
<br>
However, if a hotfix install completed during the first 24 hours,<br>
that likely means a) the hotfix did its job and the user upgraded b)<br>
the hotfix encountered a fatal error and uninstalled itself. "a" can<br>
be measured via other data sources. The ratios for "b" should<br>
hopefully be similar for the non-lost data, so we can extrapolate.<br>
<br>
High-Level Overview<br>
===================<br>
<br>
Total records encountered: > 4.5M<br>
Success rate: ~47.1%<br>
Clients with hotfix still installed: ~52.7%<br>
Uninstall due to bugs or other weirdness: 0.2%<br>
Clients experiencing download failures: 2.40%<br>
Windows compatibility mode detected: 0.70%<br>
<br>
Install Attempts<br>
================<br>
<br>
Here is a breakdown of the number of install attempts per client:<br>
<br>
0 4.96%<br>
1 92.77%<br>
2 2.12%<br>
3 0.12%<br>
<br>
Here is a breakdown of failure counts:<br>
<br>
0 52.47%<br>
1 46.68%<br>
2 0.75%<br>
3 0.08%<br>
4 0.01%<br>
<br>
So, ~47.5% of our install attempts result in failure. Here is a<br>
breakdown of the exit codes for the overall install attempts:<br>
<br>
Success 45.04%<br>
Elevation Cancelled 40.76%<br>
Accessing Log Failed 4.55%<br>
Unarchiving Failed 0.28%<br>
Elevation Failed 0.23%<br>
Installation Failed 0.08%<br>
Other 0.08%<br>
<br>
"Elevation Cancelled" means that we showed a UAC prompt to the user<br>
and the user did not grant privileges to allow the install to<br>
continue. This is treated as a transient failure by the hotfix.<br>
<br>
"Elevation Failed" means the user attempted to grant privileges via<br>
UAC but couldn't. This could mean the user needs an admin password<br>
(which they do not know or entered incorrectly).<br>
<br>
"Accessing log failed" is tracked by bug 1040234 and appears to be a<br>
legitimate issue with the hotfix, affecting mostly Windows XP users.<br>
<br>
Upgraded From<br>
=============<br>
<br>
Clients that successfully upgraded were upgraded from the following<br>
Firefox versions:<br>
<br>
10 0.41%<br>
11 0.36%<br>
12 2.01%<br>
13 0.48%<br>
14 0.47%<br>
15 0.74%<br>
16 0.85%<br>
17 0.39%<br>
18 0.36%<br>
19 0.44%<br>
20 2.13%<br>
21 3.75%<br>
22 3.89%<br>
23 4.70%<br>
24 6.29%<br>
25 7.78%<br>
26 12.47%<br>
27 16.86%<br>
28 30.71%<br>
Unknown 4.91%<br>
<br>
Having more clients come from more modern versions is expected, as<br>
Firefox users are on a "long tail" of versions.<br>
<br>
We somehow had 127 clients upgrade from the Aurora or Nightly<br>
channel (Firefox versions with "a" in the version string). We didn't<br>
explicitly check for this in the "is hotfix applicable" check.<br>
Instead, we checked that channel == "release." I suspect these are<br>
instances of profile copying.<br>
<br>
Notification<br>
============<br>
<br>
If an installation fails in a transient manner, the hotfix will show<br>
a notification on startup until Firefox is upgraded or a "real"<br>
error is encountered.<br>
<br>
Only 5.19% of reporting clients have seen the notification. This<br>
value seems extremely weird to me. We have the hotfix still<br>
installed on over 50% of the reporting clients. If we had daily<br>
users who were restarting the browser daily, I'd expect clients<br>
seeing the notification to be similar to those that still have the<br>
hotfix installed. *This needs further investigation.*<br>
<br>
Profile Sharing<br>
===============<br>
<br>
0.12% of clients had the hotfix uninstalled because it was "no<br>
longer applicable." This means the hotfix, when installed, met the<br>
conditions of the hotfix. But those conditions changed making the<br>
hotfix no longer applicable, causing the hotfix to uninstall.<br>
<br>
The overwhelming majority of these clients appeared to switch to a<br>
different release channel. The beta channel accounted for 0.03%. The<br>
"none" channel accounted for 0.08%. This means that<br>
app.update.channel is throwing or is returning "none." I'm<br>
scratching my head on this one.<br>
<br>
It's worth noting that if I turn off record deduplication, 0.16% of<br>
records are reporting "no longer applicable." I suspect a small<br>
faction of our users are copying profiles and running them with a<br>
different Firefox install. (We've seen FHR records that better<br>
identify what's going on.)<br>
<br>
Download Speed<br>
==============<br>
<br>
I performed a crude analysis of the average download speed of each<br>
client. Here is a histogram of how things broke down:<br>
<br>
Bps % cumulative %<br>
0 1.21 1.21<br>
10000 2.58 3.79<br>
20000 2.77 6.56<br>
30000 2.68 9.24<br>
40000 2.83 12.07<br>
50000 2.69 14.75<br>
60000 2.07 16.82<br>
70000 1.90 18.72<br>
80000 1.86 20.58<br>
90000 1.87 22.45<br>
100000 4.57 27.02<br>
125000 2.95 29.97<br>
150000 2.78 32.75<br>
175000 2.74 35.49<br>
200000 5.33 40.82<br>
250000 3.39 44.21<br>
300000 3.16 47.37<br>
350000 2.93 50.30<br>
400000 2.46 52.76<br>
450000 2.30 55.07<br>
500000 3.90 58.96<br>
600000 3.70 62.66<br>
700000 2.90 65.56<br>
800000 2.38 67.94<br>
900000 1.98 69.92<br>
1000000 8.03 77.95<br>
1500000 4.91 82.87<br>
2000000 5.39 88.26<br>
3000000 2.75 91.00<br>
4000000 1.66 92.66<br>
5000000 7.33 100.00<br>
<br>
"B" in these values is bytes, not bits.<br>
<br>
It's worth noting that 22.45% of hotfix users are downloading at<br>
<100KBps and 50% are below 400KBps.<br>
<br>
I empathize with the 1.21% of users in the 0-10KBps bucket: we made<br>
them download a ~28M installer which would take at least 48 minutes.<br>
Users in the other slow buckets don't fare much better. I haven't<br>
cross-referenced these aggregate results to see if our users on<br>
slower connections had the patience to sit through the full download.<br>
<br>
The hotfix did not limit the Necko channel in any way (unlike the<br>
built-in update mechanism, which throttles).<br>
<br>
Download Failures<br>
=================<br>
<br>
As reported above, 2.40% of reporting clients experienced at least 1<br>
download failure.<br>
<br>
A download failure occurs when the hotfix thinks it has successfully<br>
downloaded an installer but the file size or SHA512 verification fails.<br>
This could mean a) download/Necko code is buggy b) HTTP servers are<br>
lying or misbehaving (possibly captive portal) c) evil parties are<br>
rewriting Firefox installers. This will need additional analysis.<br>
<br>
Summary<br>
=======<br>
<br>
The hotfix has so far upgraded a few million clients to Firefox 30!<br>
<br>
I'm concerned about the few million clients that still have the<br>
hotfix installed, especially the ones that apparently haven't seen a<br>
notification. I'll need to deep dive into payloads to get a feel for<br>
what's happening.<br>
<br>
Remember, we lost data for clients that completed in the first ~24<br>
hours.<br>
<br>
Let me know if you have any questions or data requests.<br>
<br>
Gregory<br>
<br></div></div>
______________________________<u></u>___________________<br>
firefox-dev mailing list<br>
<a href="mailto:firefox-dev@mozilla.org" target="_blank">firefox-dev@mozilla.org</a> <mailto:<a href="mailto:firefox-dev@mozilla.org" target="_blank">firefox-dev@mozilla.<u></u>org</a>><br>
<a href="https://mail.mozilla.org/__listinfo/firefox-dev" target="_blank">https://mail.mozilla.org/__<u></u>listinfo/firefox-dev</a><br>
<<a href="https://mail.mozilla.org/listinfo/firefox-dev" target="_blank">https://mail.mozilla.org/<u></u>listinfo/firefox-dev</a>><br>
<br>
<br>
</blockquote>
<br>
</blockquote></div><br></div>