MozMill in Thunderbird: looking back and moving forward

Siddharth Agarwal sagarwal at mozilla.com
Sat Jul 30 15:33:23 UTC 2011


(cross-posted from 
http://monogatari.doukut.su/2011/07/mozmill-in-thunderbird-looking-back-and.html 
)

The Thunderbird team started using MozMill a couple of years ago to test 
UI interactions. From its humble beginnings 
<https://hg.mozilla.org/comm-central/pushloghtml?changeset=fa041768f27d> 
with a couple of not-really-tests to over 400 tests covering a variety 
of UI interactions with tinderbox coverage on all 3 major platforms, the 
Thunderbird MozMill test suite has matured into an important and fairly 
reliable indicator of frontend code correctness.

It's not always been like this, though -- the test harness has had 
several major issues over the time it has existed.

  * *Random test failures.* These used to be very common before we made
    a concerted effort to fix them around the beginning of last year. A
    lot of them were bugs in the tests themselves, but there were at
    least a few subtle bugs
    <https://bugzilla.mozilla.org/show_bug.cgi?id=545674> lurking in the
    code. Making predicate timeouts count as failures
    <https://bugzilla.mozilla.org/show_bug.cgi?id=540110> helped a lot
    in tightening things up. A few random failures still remain but
    they're infrequent enough that we don't have to make fixing them a
    top priority.
  * *Linux support.* We got MozMill tinderboxes for Windows and Mac up
    pretty early, but Linux tests were broken for the longest while. The
    tests ran in a different order, the wrong part of the UI ended up
    receiving any keystrokes made... it made Linux frontend developers'
    lives a pain, and it took a rather large patch
    <https://bugzilla.mozilla.org/show_bug.cgi?id=633498> from the
    wonderful asuth <http://www.visophyte.org/blog/> to finally fix
    things for good. MozMill tinderboxes for Linux were finally turned
    on some time early this year.

It's still not as good as it can be, though. Some of the issues that 
remain are

  * *IMAP, SMTP and NNTP support.* We've had fakeservers
    <https://developer.mozilla.org/en/MailNews_fakeserver> for all the
    protocols we support for a while, but because of the way they work
    they're limited to xpcshell tests for now. Thus one can't write a
    frontend test that focuses on issues with our UI with IMAP servers
    (and we have plenty of those). This should be fixed right after
    jcranmer <http://quetzalcoatal.blogspot.com/> moves the fakeservers
    out of the main process
    <http://quetzalcoatal.blogspot.com/2011/06/fakeserver-and-future-changes.html>,
    and I look forward to seeing the first IMAP tests soon.
  * *Outdated MozMill versions.* The MozMill developers have introduced
    subtle API changes over time, which means we haven't been able to
    move to more recent versions as easily as we'd hoped to. (Not
    blaming them, though, since MozMill was still pretty immature when
    we decided to start using it.)
  * *External dependencies.* Imagine you're a new developer who'd like
    to get started with your first MozMill test. You discover that you
    need to install a list of packages through |easy_install| or
    similar. You can't just |easy_install mozmill|, either: you need to
    specify the URL
    <https://developer.mozilla.org/en/Thunderbird/Thunderbird_MozMill_Testing>
    to each package you need. If you need a different version of MozMill
    for something else, you need to learn about and use virtualenv
    <http://www.virtualenv.org/>. All this is far more troublesome than
    it should be.

    Well, good news, everyone!
    <http://www.youtube.com/watch?v=1D1cap6yETA> We're soon going to
    move MozMill to within the tree
    <https://bugzilla.mozilla.org/show_bug.cgi?id=656736> as part of a
    larger patch to upgrade it to the latest version. (The move was
    spurred by the fact that we didn't want to check the requisite test
    fixes into older branches and all our branches were tested by the
    same builders, so had the same version of MozMill installed. But
    let's not focus on that. :) ) Once the patch is in the tree, you'll
    be able to run MozMill tests without installing anything else.

MozMill's future is bright: more developers will be able to write tests 
for more of our code faster and better.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/tb-planning/attachments/20110730/8ab0ff4f/attachment.html>


More information about the tb-planning mailing list