quasi-survey on writing mozmill tests for thunderbird
ben.bucksch at beonex.com
Thu Mar 17 18:39:38 UTC 2011
thanks for takling this!
My below statements are based on my experiences one year ago, and they
were very negative. So, take this with lots of salt.
On 17.03.2011 08:35, Andrew Sutherland wrote:
> - look at existing tests and use the helper methods they find in
> those? (Possibly via copying an existing test and starting from there.)
IMHO, it would be better to provide API documentation and some simple
sample tests, for various situations, but that depends on the personal
> 2) How frustrating is mozmill test development? Specifically:
> - How hard is it to get the environment set up?
Last time I checked, there were no docs, I had to compile third-party
software myself, and it kept flashing around for an hour. I hear it's
now possible to run it via VNC, but I guess that doesn't make it exactly
easy to set up, either.
Also, the test suite takes an hour or so to run. My patience ends after
2-3 minutes. No, I can't stop my work and resume later, my brain context
is gone at that point.
Even the UI to run single tests (then in Firefox) was almost
incomprehensible and produced confusing results.
The tests also created spurious failures, which made me hunt ghosts, and
it seems that problem still exists, as you wrote "Running a test on its
own tends to fail a lot less; interaction with other tests in the same
directory can complicate things...".
All in all, my last attempt was so ridiculously painful that I simply
refuse to run any tests anymore. For that reason, I plain out stopped
working on any bugs where the reviewer asked me to create tests, e.g.
"guess POP config".
> - How hard is it to debug a test?
Practically impossible for me.
> - How annoying is the length of time the tests take to run?
> 3) What would make it easier
* proper documentation, for 1) setup/run, 2) how to write tests 3)
test API, maybe including some sample code for various situations
* mozmill comes with source code, depending only on things that I
can "apt-get install" from standard repos (e.g. Ubuntu universe)
* a single commandline command (that's easy to remember and type) to
run a single test, e.g. "mozmill
* deterministic results, all tests always PASS
* FAIL only when I actually introduced a bug (not just changed
something). The current tests are not too bad with that, for UI
tests, but could be better.
> - Ability to better understand what is happening in local failures
> with logging. (Noting that we now have two options for this, but
> they're not particularly documented and require varying amounts of
> setup, both of which could potentially be somewhat mitigated.)
The above "mozmill" command to run one test should by default give
output that helps the developer maximally, i.e. just the right amount of
verbosity (report all errors with stacks, and important points that
PASS, but don't spam me).
> - Ability to develop mozmill tests by recording interactions with
> Thunderbird. (Note: this is actually a hard one, but it's worth
> knowing how desirable it is.)
If you can make that reliable, that'd be cool. But that "don't FAIL on
changes unless they introduced bugs" will be hard to satisfy in this
case. I don't want to adapt lots of tests for every change I make. You'd
not only need a recorder, but something that 1) compares SHOULD and IS
result, visually, and point out the differences 2) let me investigate
each 3) let me approve the change and it generate a changeset that
adapts the test to the IS result, in this case.
So, while being the "Holy Grail" of automated testing, and very useful,
it's a lot of work to get right, I guess there are other ways to improve
More information about the tb-planning