MozMill in Thunderbird: looking back and moving forward
sagarwal at mozilla.com
Sat Jul 30 15:33:23 UTC 2011
The Thunderbird team started using MozMill a couple of years ago to test
UI interactions. From its humble beginnings
with a couple of not-really-tests to over 400 tests covering a variety
of UI interactions with tinderbox coverage on all 3 major platforms, the
Thunderbird MozMill test suite has matured into an important and fairly
reliable indicator of frontend code correctness.
It's not always been like this, though -- the test harness has had
several major issues over the time it has existed.
* *Random test failures.* These used to be very common before we made
a concerted effort to fix them around the beginning of last year. A
lot of them were bugs in the tests themselves, but there were at
least a few subtle bugs
<https://bugzilla.mozilla.org/show_bug.cgi?id=545674> lurking in the
code. Making predicate timeouts count as failures
<https://bugzilla.mozilla.org/show_bug.cgi?id=540110> helped a lot
in tightening things up. A few random failures still remain but
they're infrequent enough that we don't have to make fixing them a
* *Linux support.* We got MozMill tinderboxes for Windows and Mac up
pretty early, but Linux tests were broken for the longest while. The
tests ran in a different order, the wrong part of the UI ended up
receiving any keystrokes made... it made Linux frontend developers'
lives a pain, and it took a rather large patch
<https://bugzilla.mozilla.org/show_bug.cgi?id=633498> from the
wonderful asuth <http://www.visophyte.org/blog/> to finally fix
things for good. MozMill tinderboxes for Linux were finally turned
on some time early this year.
It's still not as good as it can be, though. Some of the issues that
* *IMAP, SMTP and NNTP support.* We've had fakeservers
<https://developer.mozilla.org/en/MailNews_fakeserver> for all the
protocols we support for a while, but because of the way they work
they're limited to xpcshell tests for now. Thus one can't write a
frontend test that focuses on issues with our UI with IMAP servers
(and we have plenty of those). This should be fixed right after
jcranmer <http://quetzalcoatal.blogspot.com/> moves the fakeservers
out of the main process
and I look forward to seeing the first IMAP tests soon.
* *Outdated MozMill versions.* The MozMill developers have introduced
subtle API changes over time, which means we haven't been able to
move to more recent versions as easily as we'd hoped to. (Not
blaming them, though, since MozMill was still pretty immature when
we decided to start using it.)
* *External dependencies.* Imagine you're a new developer who'd like
to get started with your first MozMill test. You discover that you
need to install a list of packages through |easy_install| or
similar. You can't just |easy_install mozmill|, either: you need to
specify the URL
to each package you need. If you need a different version of MozMill
for something else, you need to learn about and use virtualenv
<http://www.virtualenv.org/>. All this is far more troublesome than
it should be.
Well, good news, everyone!
<http://www.youtube.com/watch?v=1D1cap6yETA> We're soon going to
move MozMill to within the tree
<https://bugzilla.mozilla.org/show_bug.cgi?id=656736> as part of a
larger patch to upgrade it to the latest version. (The move was
spurred by the fact that we didn't want to check the requisite test
fixes into older branches and all our branches were tested by the
same builders, so had the same version of MozMill installed. But
let's not focus on that. :) ) Once the patch is in the tree, you'll
be able to run MozMill tests without installing anything else.
MozMill's future is bright: more developers will be able to write tests
for more of our code faster and better.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the tb-planning