<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<link href="chrome://translator/skin/floatingPanel.css"
type="text/css" rel="stylesheet">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Here is the Etherpad starting point
again:<br>
<br>
<br>
<a class="moz-txt-link-freetext" href="https://etherpad.mozilla.org/qIkqbtHTSV">https://etherpad.mozilla.org/qIkqbtHTSV</a><br>
<br>
<br>
<br>
<br>
</div>
<blockquote cite="mid:5043867F.3000409@mozilla.com" type="cite">
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
Hi,<br>
<br>
tl,dr; Description of how quality control is done and ideas on how
we should continue.<br>
<br>
I'm ludovic and I'm the quality lead for Thunderbird. I've been
managing quality with the help of the community since february
2009. I work full time (5 days a week) on quality and I try to
make sure that we ship a stable product.<br>
<br>
So what does my job consist of these days :<br>
<br>
<h1>Today</h1>
<h2>I Tools</h2>
maintaining the tools and making sure the tools mozilla uses stay
compatible with the way we use them. The tools we use are :<br>
<ul>
<li>sorocco (the interface to the crash database)<br>
</li>
<li>bugzilla (the bug tracking system<br>
</li>
<li>Litmus (and it's coming replacement, the testcase management
software)</li>
</ul>
There's not much work there once in a while make sure things
continue to work - and the people maintaining the tools always
have been a pleasure to work with.<br>
<br>
<h2>II Testing</h2>
<p>Testing is divided between new feature testing and regression
testing.<br>
</p>
<h3>New feature testing</h3>
This involves figuring out what the new feature will be , how it
will work and use it with normal use case and then edge use cases.
Once testing is done this means :<br>
<ul>
<li>Adding new testcase to Litmus or testcase management tool</li>
<li>Filling bugs so the developers can try to fix them before
the features make it to a release</li>
</ul>
<p>I usually send emails to the Thunderbird-testers mailing list
when I feel that the feature is "ready" and to get broader
testing - in this email I usually explain what the new feature
is suppose to do, and give hints on where to file bugs.<br>
</p>
<h3>Regression testing<br>
</h3>
This is done prior to releases and is what I do to make sure that
Thunderbird doesn't regress. The auto update mechanism I always
test myself for all releases - as it's easier and needs
coordination with Release Engineering and Release drivers, this is
around 2 hours of testing when everything is coordinated properly.
<br>
<br>
Since we've jumped on the Rapid release trains, I usually do my
regression testing in the following way :<br>
<ol>
<li>Fire my windows VM</li>
<li>Install the beta that we are going to release, of final
build</li>
<li>Create a pop Account</li>
<ol>
<li>Receive emails</li>
<li>Send emails</li>
<li>send email to the other account with return receipt on</li>
<li>create a saved search</li>
</ol>
<li>Create an imap Account</li>
<ul>
<li>do the same as pop3 testing</li>
</ul>
<li>Create a RSS account</li>
<ul>
<li>Add a few feeds , read them (and as I usually use a
newspaper feeds see that it get's updated)</li>
</ul>
<li>Create a NNTP account</li>
<ul>
<li>Post to some .test newsgroup</li>
</ul>
<li>send an email to release driver stating that the build is ok
or not</li>
</ol>
When a large feature lands I will call for testers on
mozilla.dev.apps.thunderbird, tb-plaining and the
thunderbird-testers mailing list (eg last time we did that was
when maildir-like support landed), to have more than a pair of
eyes trying to figure out what might be broken. I don't do it more
often because I feel it's time consuming and I don't get many
testers (eg max is 20, min is 1 or 2) and we don't find
regressions , or not enough to my taste when using this method.<br>
<br>
<h2>III Bugzilla</h2>
Most of my time is spent in bugzilla :<br>
<ol>
<li>Reading all comments on all bugs</li>
<li>Making sure flags are set</li>
<li>Trying to recruit new contributors</li>
<ul>
<li>by email</li>
</ul>
<li>engaging with new contributors</li>
<ul>
<li>by email</li>
</ul>
<li>engaging with the rest of mozilla when we are affected by
Core issue</li>
<ul>
<li>either by mail, or in the bugs</li>
</ul>
<li>Trying to find important regression issues</li>
<ul>
<li>pinging proper developers on these</li>
</ul>
<ul>
<li>asking on irc for other issues when I'm unsure</li>
</ul>
<li>Replying ASAP to new bugs to try to capture as much
information as possible</li>
<ul>
<li>Replying on the day, the bug is fresh and bug reporters
are most likely to add and answer your questions</li>
</ul>
</ol>
<p><br>
</p>
<h1>Going Forward</h1>
<h2>I Bugzilla</h2>
<p>We need to find a process that works for both developers and
people involved in QA so that bugs get fixed.<br>
We need to fix old bugs as well as new bugs that arise from new
features landing or from Core Gecko changes.<br>
</p>
<p>Here is the easy list of criteria we should use for bubbling up
bugs :<br>
</p>
<ol>
<li>Number of people affected (we'll probably need some input
from support for this)</li>
<li>Is it because of a new feature ?</li>
<li>Is it a main feature of the product (eg an edge case of
printing)</li>
</ol>
<p>Then we'll need a way to expose those bugs/ issues to devs and
devs will need a way to look/assign and fix these. I'm thinking
about sending a summary email on a know occurence (eg once a
month, a week , every 15 days) ?<br>
</p>
I think that once we've got a list of criteria that both devs and
contributors to quality agree , we'll just need to have more
people helping in bugzilla. <br>
<br>
Right now there are between 0 and 7 people helping at various
levels in bugzilla (some searching for duplicates, some moving to
the proper component, some asking question and trying to get more
information from our users, some closing bugs that we can't do
much with - because of lack of precise information). While I've
been trying over the last few years to grow the number of
contributors in bugzilla - I've never managed to have it grow.
People come , stay and leave except a few exceptions. So if you
have ideas on how we could manage to grow the number of people
caring with bugzilla please chime in.<br>
<h2>II Testing</h2>
<p>With the time That I'll be allowed to work on the project I
should be able to continue doing update testing - but I should
probably write somewhere exactly what I'm doing so someone could
take over.<br>
</p>
<p>For new feature testing I proposed to crowd source them
directly, when a new feature lands :<br>
</p>
<ul>
<li>Organize a few days of testing where</li>
<li>people testing meet on irc</li>
<li>the developer is on irc too so he can quickly answer</li>
<li>bugs are flagged to be easily findable by any developer who
would want to fix the feature before it reaches mainstream</li>
</ul>
<p>for regression testing see below<br>
</p>
<h3>Automated testing<br>
</h3>
<p>One thing that changed the quality of Thunderbird in the last
few years was when in 2010 we forced each new patch to come with
new tests. I can't stress out how much this has made catching
regression easier and faster. I would stress that we enforce
this policy in the future in a stricter ways than it has been in
the past. <br>
</p>
<p>Also some areas are under tested by unit test I think it would
be wise to have some of our engineering effort put into adding
more tests. I've been a total failure at that in the last 3
years. Ideas on having people spending coding efforts on adding
more test are more than welcome. We could use jcranmer great
work on code caverage to easily figure out where more tests are
needed.<br>
</p>
<h3>Unformal testing</h3>
<p>This is what I call testing done by people using Early bird,
beta and daily on a a daily basis - people that use our
pre-release software all the time not just when a test event
comes up. I think it's way too fragmented right now. We get good
feedback on daily and sometimes on beta. I can't recall that we
ever caught or got any feedback on early bird. We currently
don't have enough user using pre-released version of Thunderbird
as in the pst we caught major regressions the day of the release
- this is widely due to all the configuration option mailnews
offers, plus the zillion configuration of add-ons and external
software our user base uses. <br>
Mark Banner as the complete numbers (mark can you chime these in
of the conversation please), but I think we need to grow these
numbers and maybe scatter them less (eg have more daily users
and beta users, push less earlybrid). Ideas on how to achieve
this growth , or where it should grow are very welcomed.<br>
</p>
<h3>More Formal testing</h3>
<p>I'd like to do more format testing, have a week in the beta
cycles where we gather a big number of volunteers and they run
all the testcases we have in our testcase management tool. As
said before I thought that it was time consuming to do these,
but in a context where I follow bugzilla less closely the
advantages of having such event driven testing pop up :<br>
</p>
<ul>
<li>a tracking bug so bugs are easy to find by devs.</li>
<li>we get people to test all areas (not just those used by
regular betatesters)</li>
</ul>
<p>The issue with this are :<br>
</p>
<ul>
<li>Having enough people willing to spend enough time to get
100% coverage (with 20 people participating a good hour I
think it's achievable)</li>
<li>Some test are difficult to setup (eg you need LDAP, you need
a proxy)</li>
<li>People tend to come to one event, if we have them to often
they don't come anymore</li>
<li>How to get enough people to reach 100% coverage everytime</li>
<li>Having a set of testcase that are up to date.<br>
</li>
</ul>
<p>Getting more people for a one time shot is easy. keeping them
is hard.<br>
</p>
<h2>III Tools</h2>
<p><br>
I don't see any forthcoming issues with tools - so I'm not
really afraid on that front.<br>
</p>
<p><br>
</p>
<h2>Conclusion</h2>
<p>None yet, chime in. Talk, argue and let's build a plan on how
to make Thunderbird even better and raise the quality standard
for email clients.<br>
</p>
<p>Ludo<br>
ps I probably forgot a few things here, please ask, argue and
let's get Quality forward.<br>
<br>
</p>
<pre class="moz-signature" cols="72">--
@lhirlimann on twitter
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://wiki.mozilla.org/Thunderbird:Testing">https://wiki.mozilla.org/Thunderbird:Testing</a>
my photos <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.flickr.com/photos/lhirlimann/collections/">http://www.flickr.com/photos/lhirlimann/collections/</a></pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
tb-planning mailing list
<a class="moz-txt-link-abbreviated" href="mailto:tb-planning@mozilla.org">tb-planning@mozilla.org</a>
<a class="moz-txt-link-freetext" href="https://mail.mozilla.org/listinfo/tb-planning">https://mail.mozilla.org/listinfo/tb-planning</a>
</pre>
</blockquote>
<br>
<div id="smartTemplate4-template">Hi <a class="moz-txt-link-abbreviated" href="mailto:tb-planning@mozilla.org">tb-planning@mozilla.org</a>,<br>
</div>
<div style="bottom: auto; left: 191px; right: auto; top: 25px;"
class="translator-theme-default" id="translator-floating-panel">
<div title="Click to translate"
id="translator-floating-panel-button"></div>
</div>
</body>
</html>