Breaking url-bar search

William Pietri william at scissor.com
Fri Aug 16 16:13:01 UTC 2013


Hi, Gavin. Sorry if this is coming across as hostile; I'm not trying to 
pick on you.

It's my long-held belief that if there are problems in a software 
organization, they're systemic; individuals are doing the best they can.

On 08/16/2013 08:20 AM, Gavin Sharp wrote:
> On Fri, Aug 16, 2013 at 3:15 AM, William Pietri <william at scissor.com> wrote:
>> Ok. So it is fair to say that no data was used in estimating the impact of
>> removing this feature?
> No, I don't think that's a fair characterization, though it depends
> somewhat on what kind of "data" you mean. Experience shipping
> software, insight from user research (see e.g.
> https://blog.mozilla.org/ux/2012/12/distributed-qualitative-analysis-for-the-firefox-behavioral-segmentation-study/),
> and expertise combined with anecdotal data can often be surprisingly
> sufficient for making the right decisions. But these things are very
> frequently discounted entirely, because they always involve some
> degree of subjectivity. They can also be subject to bias, but probably
> not more so than "hard" data (or our interpretation of it).

I agree that when intuition is what you've got, that's what you should 
go with. But in this case, by "data" I mean what is traditionally meant 
by the word: reliable numbers.

So in this case, it sounds like there was no data available to you, so 
you made an expert judgment call. Is that fair to say?


>> Also, as to the approximate number of users affected, it sounds like you
>> estimated it in the range of "a lot of people", but that could be anywhere
>> from thousands to millions? Could it be tens of millions?
> It seems unlikely that you'll trust my judgement here, so I won't try
> to convince you, but I think it quite unlikely that this had a
> negative impact on millions of users. We do not know for sure, of
> course.

Well, it's nothing personal against your judgment; I'm sure it's quite 
good, especially working from no real data on this. I'm just trying to 
put error bars on the number of affected users.

One way to work at this is to work backwards from impact. So if you work 
this equation:

    # of users who use this feature *
    % of users who have upgraded to FF23 *
    % of upgraders who consciously recognize the breakage *
    % of recognizers who bother to search *
    % of searchers who find what happened *
    % of those people who find the add-on *
    % of people who bother to install the add-on =
    538

You can try your own percentages, and they'll probably be more correct 
than mine, but when I play with this I get pretty large numbers.

So from that, it seems fair to say that the number of affected users is 
thousands to millions, but tens of millions is out of bounds, yes?

>> But I'm puzzled by what you write. I thought the theory of this feature was
>> that people with hijacked url bar search needed to control that so much that
>> it was worth killing a feature. But if most people can't discover the search
>> bar dropdown, how is this solving the problem?
> The feature was beneficial in two ways:
> - it consolidated search settings and removed the use of a preference
> (keyword.URL) that we have evidence was being widely abused. This is
> the primary source of hijacking-protection.
> - because the search settings were consolidated and are all controlled
> by user-visible UI (the search bar), in any remaining cases where a
> third party changes your search settings non-maliciously, users now
> have the ability to easily revert it without having to resort to tools
> like https://addons.mozilla.org/en-US/firefox/addon/searchreset/
>
> Lack of discoverability of the search bar dropdown only negatively
> impacts that second benefit, and while it might mitigate it somewhat,
> overall the impact was still positive (some users do know how to use
> the dropdown).
>

Could you help me understand how the first benefit is different than the 
second? I'm not seeing any lasting user benefit to point 1 on its own; 
hijackers will presumably quickly retarget.

William
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20130816/acbb94df/attachment.html>


More information about the firefox-dev mailing list