How I measured your password manager

Ryan Feeley rfeeley at mozilla.com
Tue Mar 31 17:53:13 UTC 2015


Hi team: Here are some ideas for success metrics. I would love to get feedback on the ideas and then turn them into actionable bugs and eventually a dashboard. What can we know?

Happiness – Measures of user attitudes, often collected via survey. For example: satisfaction, perceived ease of use, and net-promoter score.
We want users to only remove the logins they intend to remove.
They don’t add a login back right after they remove it. Don't click Undo.
We want users to be less frustrated by duplicate passwords.
Users have fewer duplicate entries for sites.
Engagement – Level of user involvement, typically measured via behavioral proxies such as frequency, intensity, or depth of interaction over some time period. Examples might include the number of visits per user per week or the number of photos uploaded per user per day.
Showing the key icon more often as a door hanger icon means users will manage their logins more.
Counts of clicking on key icon, viewing passwords, editing passwords, copying usernames and passwords
Showing a link to the Saved Logins at the bottom of the auto-complete login will lead to increased engagement.
Clicking on Saved Passwords row, viewing passwords, editing passwords, copying usernames and passwords
Showing the key icon when on a login form means users will be able to access needed credentials.
Clicking on the blue key icon, retrieve saved logins, click copy username/password, and use login button
Adoption – New users of a product or feature. For example: the number of accounts created in the last seven days or the percentage of Gmail users who use labels.
We want people to sense that our password manager is secure and trustworthy.
Users don’t install 3rd party password managers
Users check (or leave checked) to remember passwords for sites
We believe the new door hanger design will lead to increased engagement.
Saving more passwords, editing more passwords, never remembering more passwords
Retention – The rate at which existing users are returning. For example: how many of the active users from a given time period are still present in some later time period? You may be more interested in failure to retain, commonly known as “churn.”
Users trust Firefox to remember their password.
Users check (or leave checked) to remember passwords for sites
http://telemetry.mozilla.org/#filter=nightly%2F39%2FPWMGR_SAVING_ENABLED%2Fsaved_session%2FFirefox&aggregates=multiselect-all!Submissions&evoOver=Builds&locked=true&sanitize=false&renderhistogram=Graph <http://telemetry.mozilla.org/#filter=nightly/39/PWMGR_SAVING_ENABLED/saved_session/Firefox&aggregates=multiselect-all!Submissions&evoOver=Builds&locked=true&sanitize=false&renderhistogram=Graph>

Task Success – This includes traditional behavioral metrics of user experience, such as efficiency (e.g. time to complete a task), effectiveness (e.g. percent of tasks completed), and error rate. This category is most applicable to areas of your product that are very task-focused, such as search or an upload flow.
We want users to find logins as needed so they can authenticate to websites with little effort.
After user clicks Use Login they don’t immediately interact with the saved logins in another way.
Users are faster than users who are not saving passwords.
Users will save fewer incorrect logins because they are more readable and editable.
Fewer password resets on the web.
Resetting password less often.

Ryan Feeley
UX, Cloud Services
Mozilla UX
IRC: rfeeley

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/passwords-dev/attachments/20150331/b768aa5f/attachment.html>


More information about the Passwords-dev mailing list