Breaking url-bar search

Tyler Downer tdowner at mozilla.com
Fri Aug 16 16:34:55 UTC 2013


Replies inline

Tyler Downer
User Advocate for All Things Firefoxy

On 8/16/13 9:13 AM, William Pietri wrote:
> Hi, Gavin. Sorry if this is coming across as hostile; I'm not trying 
> to pick on you.
>
> It's my long-held belief that if there are problems in a software 
> organization, they're systemic; individuals are doing the best they can.
>
> On 08/16/2013 08:20 AM, Gavin Sharp wrote:
>> On Fri, Aug 16, 2013 at 3:15 AM, William Pietri<william at scissor.com>  wrote:
>>> Ok. So it is fair to say that no data was used in estimating the impact of
>>> removing this feature?
>> No, I don't think that's a fair characterization, though it depends
>> somewhat on what kind of "data" you mean. Experience shipping
>> software, insight from user research (see e.g.
>> https://blog.mozilla.org/ux/2012/12/distributed-qualitative-analysis-for-the-firefox-behavioral-segmentation-study/),
>> and expertise combined with anecdotal data can often be surprisingly
>> sufficient for making the right decisions. But these things are very
>> frequently discounted entirely, because they always involve some
>> degree of subjectivity. They can also be subject to bias, but probably
>> not more so than "hard" data (or our interpretation of it).
>
> I agree that when intuition is what you've got, that's what you should 
> go with. But in this case, by "data" I mean what is traditionally 
> meant by the word: reliable numbers.
>
> So in this case, it sounds like there was no data available to you, so 
> you made an expert judgment call. Is that fair to say?
No it isn't. This change was the first part of a series of fixes by 
Project Squeaky, https://wiki.mozilla.org/AMO/Squeaky, in an initiative 
to harden Firefox against malicious and Adware Add-ons, which are a very 
severe problem for millions of Firefox users. We didn't just pull this 
change out of anecdotal data alone, but it is the result of a lot of 
research by the User Advocacy and AMO Teams. I can't share the data in a 
public list, but we do know that Malicious add-ons are a very real 
problem for users, and often the most visible impact of add-on hijacking 
is that the keyword.url has been changed from default, for two reasons:
1. Every malicious add-on we have seen changes this pref.
2. There is no visible UI to change this pref, meaning that very few 
users would actually have changed it on their own. Plus, users who have 
had their keyword.url hijacked have no visible way to reset it.

This the decision to consolidate search prefs, to both remove an attack 
vector and to make it easier for hijacked users to Reset their prefs to 
whatever they want.
>
>
>>> Also, as to the approximate number of users affected, it sounds like you
>>> estimated it in the range of "a lot of people", but that could be anywhere
>>> from thousands to millions? Could it be tens of millions?
>> It seems unlikely that you'll trust my judgement here, so I won't try
>> to convince you, but I think it quite unlikely that this had a
>> negative impact on millions of users. We do not know for sure, of
>> course.
>
> Well, it's nothing personal against your judgment; I'm sure it's quite 
> good, especially working from no real data on this. I'm just trying to 
> put error bars on the number of affected users.
>
> One way to work at this is to work backwards from impact. So if you 
> work this equation:
>
>     # of users who use this feature *
>     % of users who have upgraded to FF23 *
>     % of upgraders who consciously recognize the breakage *
>     % of recognizers who bother to search *
>     % of searchers who find what happened *
>     % of those people who find the add-on *
>     % of people who bother to install the add-on =
>     538
>
> You can try your own percentages, and they'll probably be more correct 
> than mine, but when I play with this I get pretty large numbers.
>
> So from that, it seems fair to say that the number of affected users 
> is thousands to millions, but tens of millions is out of bounds, yes?
>
It depends on how you define "affected". If you mean "Users who had 
their keyword.url hijacked and now have had it reset" then yes, lots of 
users have been affected, mostly positively. If you mean "Users who want 
multiple search engines and now have to install an add-on to restore 
that functionality", then you are talking thousands of users at the most.
>>> But I'm puzzled by what you write. I thought the theory of this feature was
>>> that people with hijacked url bar search needed to control that so much that
>>> it was worth killing a feature. But if most people can't discover the search
>>> bar dropdown, how is this solving the problem?
>> The feature was beneficial in two ways:
>> - it consolidated search settings and removed the use of a preference
>> (keyword.URL) that we have evidence was being widely abused. This is
>> the primary source of hijacking-protection.
>> - because the search settings were consolidated and are all controlled
>> by user-visible UI (the search bar), in any remaining cases where a
>> third party changes your search settings non-maliciously, users now
>> have the ability to easily revert it without having to resort to tools
>> likehttps://addons.mozilla.org/en-US/firefox/addon/searchreset/
>>
>> Lack of discoverability of the search bar dropdown only negatively
>> impacts that second benefit, and while it might mitigate it somewhat,
>> overall the impact was still positive (some users do know how to use
>> the dropdown).
>>
>
> Could you help me understand how the first benefit is different than 
> the second? I'm not seeing any lasting user benefit to point 1 on its 
> own; hijackers will presumably quickly retarget.
The search dropdown is at least more discoverable than about:config, so 
we made the story better. Also, don't think this is the end of project 
squeaky's efforts to harden Firefox, we've got more plans in the works 
and you can find them all from the wiki.
>
> William
>
>
> _______________________________________________
> firefox-dev mailing list
> firefox-dev at mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20130816/c4b0de8a/attachment.html>


More information about the firefox-dev mailing list