quasi-survey on writing mozmill tests for thunderbird
Pidgeot18 at verizon.net
Thu Mar 17 14:12:04 UTC 2011
On 03/17/2011 03:35 AM, Andrew Sutherland wrote:
> I'm interested in getting some feedback about the current pain level
> and methodology people are using to write thunderbird mozmill tests.
> (This is to improve things as opposed to tricking people into outing
> themselves for not having read every file in our large code-base :)
I haven't written a lot of mozmill tests, but I'll answer what I can...
> 1) Knowing what helper methods to use. Do people:
> - look at existing tests and use the helper methods they find in
> those? (Possibly via copying an existing test and starting from there.)
Via the "copy an existing test and modify the hell out of it" (granted,
I was trying to hook up fakeserver to Mozmill).
> 2) How frustrating is mozmill test development? Specifically:
> - How hard is it to get the environment set up?
I had some problem here, although I think I was trying to do something
not recommended. There is also an issue that, if you build lightning as
well, then you get the Lightning migration dialog popping up first until
you decide to uninstall the lightning extension.
> - How hard is it to write a test that passes, even if you are just
> running a single test locally on your box. (Running a test on its own
> tends to fail a lot less; interaction with other tests in the same
> directory can complicate things...)
SOLO_TEST versus SOLO_FILE caused me much pain when debugging,
especially since the framework gives you only a very poor indication
that there is no test file actually being run.
> - How hard is it to debug a test?
IIRC, some exception failures weren't properly caught by mozmill, which
required lots of dump statements and try/catch blocks to actually figure
out what was going on.
> - How annoying is the length of time the tests take to run?
It's not the length of time to run the tests that bothers me, it's how
long it took after the main window pops up before the test started running.
> 3) What would make it easier, with degree of preference:
> - Just extracting web-viewable (non-sourcey) docs without further
> categorization from what we have in the helpers.
> - The docs from above, but grouped by what it is they do.
> - Tutorial-style docs.
> - Ability to better understand what is happening in local failures
> with logging. (Noting that we now have two options for this, but
> they're not particularly documented and require varying amounts of
> setup, both of which could potentially be somewhat mitigated.)
> - Ability to develop mozmill tests by recording interactions with
> Thunderbird. (Note: this is actually a hard one, but it's worth
> knowing how desirable it is.)
> - other stuff
I don't write anywhere near enough tests to be able to adequately
respond here, especially since what I want to write is moderately
orthogonal to probably most mozmill tests (tests that require UI to work
properly, but aren't strictly UI-driven actions).
> 4) Any other thoughts about the mozmill test development setup.
It would be swell if we  could integrate the fake servers into mozmill.
As a postlude background: my first and so far only experience with
writing mozmill tests is in trying to verify the correctness of the news
URI handling, as I discovered a few regressions in the command-line URI
handler for TB. (Actually, I'm not sure how things manage to work right
now, which really confuses me, but that's beside the point). This means
that a large portion of my work was spent trying to figure out how to
get the fakeserver stuff to truly work with mozmill... which doesn't
work so well, as you might imagine.
 By "we", I of course mean "someone other than myself". :-P
Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald E. Knuth
More information about the tb-planning