quasi-survey on writing mozmill tests for thunderbird

Ben Bucksch ben.bucksch at beonex.com
Thu Mar 17 18:39:38 UTC 2011

Hey Andrew,

thanks for takling this!

My below statements are based on my experiences one year ago, and they 
were very negative. So, take this with lots of salt.

On 17.03.2011 08:35, Andrew Sutherland wrote:
> - look at existing tests and use the helper methods they find in 
> those?  (Possibly via copying an existing test and starting from there.)

IMHO, it would be better to provide API documentation and some simple 
sample tests, for various situations, but that depends on the personal 

> 2) How frustrating is mozmill test development?  Specifically:
> - How hard is it to get the environment set up?

Extremely frustrating.

Last time I checked, there were no docs, I had to compile third-party 
software myself, and it kept flashing around for an hour. I hear it's 
now possible to run it via VNC, but I guess that doesn't make it exactly 
easy to set up, either.

Also, the test suite takes an hour or so to run. My patience ends after 
2-3 minutes. No, I can't stop my work and resume later, my brain context 
is gone at that point.

Even the UI to run single tests (then in Firefox) was almost 
incomprehensible and produced confusing results.

The tests also created spurious failures, which made me hunt ghosts, and 
it seems that problem still exists, as you wrote "Running a test on its 
own tends to fail a lot less; interaction with other tests in the same 
directory can complicate things...".

All in all, my last attempt was so ridiculously painful that I simply 
refuse to run any tests anymore. For that reason, I plain out stopped 
working on any bugs where the reviewer asked me to create tests, e.g. 
"guess POP config".

> - How hard is it to debug a test?

Practically impossible for me.

> - How annoying is the length of time the tests take to run?


> 3) What would make it easier

    * proper documentation, for 1) setup/run, 2) how to write tests 3)
      test API, maybe including some sample code for various situations
    * mozmill comes with source code, depending only on things that I
      can "apt-get install" from standard repos (e.g. Ubuntu universe)
    * a single commandline command (that's easy to remember and type) to
      run a single test, e.g. "mozmill
    * deterministic results, all tests always PASS
    * FAIL only when I actually introduced a bug (not just changed
      something). The current tests are not too bad with that, for UI
      tests, but could be better.

> - Ability to better understand what is happening in local failures 
> with logging.  (Noting that we now have two options for this, but 
> they're not particularly documented and require varying amounts of 
> setup, both of which could potentially be somewhat mitigated.)

The above "mozmill" command to run one test should by default give 
output that helps the developer maximally, i.e. just the right amount of 
verbosity (report all errors with stacks, and important points that 
PASS, but don't spam me).

> - Ability to develop mozmill tests by recording interactions with 
> Thunderbird.  (Note: this is actually a hard one, but it's worth 
> knowing how desirable it is.)

If you can make that reliable, that'd be cool. But that "don't FAIL on 
changes unless they introduced bugs" will be hard to satisfy in this 
case. I don't want to adapt lots of tests for every change I make. You'd 
not only need a recorder, but something that 1) compares SHOULD and IS 
result, visually, and point out the differences 2) let me investigate 
each 3) let me approve the change and it generate a changeset that 
adapts the test to the IS result, in this case.
So, while being the "Holy Grail" of automated testing, and very useful, 
it's a lot of work to get right, I guess there are other ways to improve 


More information about the tb-planning mailing list