<p dir="ltr">One question that I would like to see on this list is whether people's behavior changes after switching to the new theme, i.e., do they manage to adapt their known workflows and mussel memory through the transition after learning about the new ways of doing things. That requires being able to correlate between different telemetry submissions of course which may not be easy. </p>
<p dir="ltr">Cheers, <br>
Ehsan </p>
<div class="gmail_quote">On 2013-10-16 10:13 PM, "Blake Winton" <<a href="mailto:bwinton@mozilla.com">bwinton@mozilla.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Hello firefox-dev-ians,<div><br></div><div>I've been chatting with some people on the Firefox UX team, and we have some questions about the impact the new Australis re-design will have on our users. I mean, we've put a lot of work into the design of Australis, and we have some goals for how we want it to change user behaviour, but we don't <i>know</i> whether people will react to it in the way we expect. I also believe that the engineers who have been (and are still) putting a ton of work into landing the new design in a performant way are interested in how their hard work is impacting the people who use Firefox. Finally, I believe that with a set of Telemetry probes, we can get the answers to most of those questions.</div>
<div><br></div><div>The <a href="https://firefox-ux.etherpad.mozilla.org/measuringaustralis" target="_blank">list of questions</a> we have come up with so far is:</div><div><br></div><div><div style="padding-right:1px;line-height:17px">
<ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do people use the Menu Panel?</span></li></ul></div><div style="padding-right:1px;line-height:17px">
<ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">vs. how many people used the orange button?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">What do people use in the Menu Panel?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do they use new tools that they didn't use before</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">do rates of usage increase/decrease vs. pre-australis</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do people enter Customize Mode?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do people actually customize things?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">What do they add/remove?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">How many people click "Restore Defaults"?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">How do people exit Customize Mode?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do people get stuck in customize mode?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do people use the Bookmark Star?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Are they bookmarking more or less?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do they use the Bookmark dropdown?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">any change in usage across the different ways of getting at bookmarks?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">What do users do upon seeing the First Run page?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do users use what we're showing them on the First Run page?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do people have more tabs?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Do people switch between tabs more?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Can people find the right background tab now that they're deemphasized?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">How do the answers to all of the above questions change after a week? A month?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Are people using find-in-page more or less since Australis?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Ditto for everything else in the panel.</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Are the tab areas discriminable?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">How often do people click on a tab, and then click on a different one?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="background-color:transparent"><span style="padding-top:1px;padding-bottom:1px">Usage increase or decrease of the Menu (OS X</span><span style="padding-top:1px;padding-bottom:1px"> and Linux! ;</span><span style="padding-top:1px;padding-bottom:1px">)</span></span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Are people pleased with the new look and feel?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Does this have any effect on add-ons?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Will it break people?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Where the [redacted] is the add-on bar?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Are people dragging windows more or less? (We're reducing titlebar space…)</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Does the rate of people detaching tabs change?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Are people more angry with us/Firefox?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">We can use Input/Feedback for this.</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">Are people using more lightweight themes?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 1.5em"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">How do people like the new style?</span></li>
</ul></div><div style="padding-right:1px;line-height:17px"><ul style="padding:0px;margin:0px 0px 0px 3em;list-style-type:circle"><li style="padding:0px;margin:0px"><span style="padding-top:1px;padding-bottom:1px;background-color:transparent">This probably needs a survey, which isn't really Telemetry…</span></li>
</ul></div></div><div><br></div><div>I suspect there are more things we are interested in, and that some of these things aren't able to be measured through Telemetry, so any comments or additions are more than welcome!</div>
<div><br></div><div>So, my requests to you are as follows:</div><div><ol><li>What am I missing in that list that we want to know or are interested in?</li><li>Which items can't we get data for with Telemetry? (Bonus points if you can suggest an alternate way of answering the question!)</li>
<li>For the items we can measure with Telemetry, which probes do we need to add?</li><li>Am I totally off-base here? Does this sound reasonable to everyone? Does it make sense at all? (Maybe this should be #1… ;)</li></ol>
</div><div><br></div><div>Assuming this is something that we all feel is valuable, then once we have a reasonably final list, I'll be happy to file bugs and meta-bugs, and (once we have some data) talk to Metrics to figure out what the data is showing us. Heck, if it's not too hard to figure out, I'ld even be up for <a href="https://developer.mozilla.org/en-US/docs/Performance/Adding_a_new_Telemetry_probe" target="_blank">writing some of the probes</a>, but I suspect that there's more to do than I can do by myself, and so any assistance would be welcome!</div>
<div><div><br></div><div>Thank you all for your input,</div><div>Blake.</div></div><div>
<font color="#7a7a7a"><span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<div>-- </div><div>Blake Winton UX Engineer</div><div><a href="mailto:bwinton@mozilla.com" target="_blank">bwinton@mozilla.com</a></div></div></span></div></span></div></span>
</font></div>
<br></div><br>_______________________________________________<br>
firefox-dev mailing list<br>
<a href="mailto:firefox-dev@mozilla.org">firefox-dev@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/firefox-dev" target="_blank">https://mail.mozilla.org/listinfo/firefox-dev</a><br>
<br></blockquote></div>