<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
(cross-posted from
<a class="moz-txt-link-freetext" href="http://monogatari.doukut.su/2011/07/mozmill-in-thunderbird-looking-back-and.html">http://monogatari.doukut.su/2011/07/mozmill-in-thunderbird-looking-back-and.html</a>
)<br>
<br>
The Thunderbird team started using MozMill a couple of years ago to
test UI interactions. From its <a
href="https://hg.mozilla.org/comm-central/pushloghtml?changeset=fa041768f27d">humble
beginnings</a> with a couple of not-really-tests to over 400 tests
covering a variety of UI interactions with tinderbox coverage on all
3 major platforms, the Thunderbird MozMill test suite has matured
into an important and fairly reliable indicator of frontend code
correctness.<br>
<br>
It's not always been like this, though -- the test harness has had
several major issues over the time it has existed.<br>
<ul>
<li><b>Random test failures.</b> These used to be very common
before we made a concerted effort to fix them around the
beginning of last year. A lot of them were bugs in the tests
themselves, but there were at least a few <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=545674">subtle
bugs</a> lurking in the code. <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=540110">Making
predicate timeouts count as failures</a> helped a lot in
tightening things up. A few random failures still remain but
they're infrequent enough that we don't have to make fixing them
a top priority.</li>
<li><b>Linux support.</b> We got MozMill tinderboxes for Windows
and Mac up pretty early, but Linux tests were broken for the
longest while. The tests ran in a different order, the wrong
part of the UI ended up receiving any keystrokes made... it made
Linux frontend developers' lives a pain, and it took <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=633498">a
rather large patch</a> from the wonderful <a
href="http://www.visophyte.org/blog/">asuth</a> to finally fix
things for good. MozMill tinderboxes for Linux were finally
turned on some time early this year.</li>
</ul>
It's still not as good as it can be, though. Some of the issues that
remain are<br>
<ul>
<li><b>IMAP, SMTP and NNTP support.</b> We've had <a
href="https://developer.mozilla.org/en/MailNews_fakeserver">fakeservers</a>
for all the protocols we support for a while, but because of the
way they work they're limited to xpcshell tests for now. Thus
one can't write a frontend test that focuses on issues with our
UI with IMAP servers (and we have plenty of those). This should
be fixed right after <a
href="http://quetzalcoatal.blogspot.com/">jcranmer</a> moves
the fakeservers <a
href="http://quetzalcoatal.blogspot.com/2011/06/fakeserver-and-future-changes.html">out
of the main process</a>, and I look forward to seeing the
first IMAP tests soon.</li>
<li><b>Outdated MozMill versions.</b> The MozMill developers have
introduced subtle API changes over time, which means we haven't
been able to move to more recent versions as easily as we'd
hoped to. (Not blaming them, though, since MozMill was still
pretty immature when we decided to start using it.)</li>
<li><b>External dependencies.</b> Imagine you're a new developer
who'd like to get started with your first MozMill test. You
discover that you need to install a list of packages through <code>easy_install</code>
or similar. You can't just <code>easy_install mozmill</code>,
either: you need to <a
href="https://developer.mozilla.org/en/Thunderbird/Thunderbird_MozMill_Testing">specify
the URL</a> to each package you need. If you need a different
version of MozMill for something else, you need to learn about
and use <a href="http://www.virtualenv.org/">virtualenv</a>.
All this is far more troublesome than it should be.<br>
<br>
Well, <a href="http://www.youtube.com/watch?v=1D1cap6yETA">good
news, everyone!</a> We're soon going to <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=656736">move
MozMill to within the tree</a> as part of a larger patch to
upgrade it to the latest version. (The move was spurred by the
fact that we didn't want to check the requisite test fixes into
older branches and all our branches were tested by the same
builders, so had the same version of MozMill installed. But
let's not focus on that. :) ) Once the patch is in the tree,
you'll be able to run MozMill tests without installing anything
else.</li>
</ul>
MozMill's future is bright: more developers will be able to write
tests for more of our code faster and better.
</body>
</html>