quasi-survey on writing mozmill tests for thunderbird
mbanner at mozillamessaging.com
Thu Mar 17 14:40:18 UTC 2011
On 17/03/2011 07:35, Andrew Sutherland wrote:
> 1) Knowing what helper methods to use. Do people:
> - look at existing tests and use the helper methods they find in
> those? (Possibly via copying an existing test and starting from there.)
Yep, very much copy test and paste, if I know there's one that remotely
covers what I need.
> - look at the code and the jsdoc-style documentation found in the
> files in the shared-modules directory, such as
> test-folder-display-helpers and test-window-helpers.js?
Occasionally, but its not always easy to find in the massive files.
> - find that the only official-ish documentation on mozmill tests is
> for mozmill proper and not the thunderbird helpers, and so end up
> using the core mozmill helpers, at least until review time at which
> they are pointed at the wonderful world of
> test-folder-display-helpers.js and friends?
I know enough not to do that, obviously others may not.
> 2) How frustrating is mozmill test development? Specifically:
> - How hard is it to get the environment set up?
Personally I think following our instructions is easy, but the real
issue comes with mozmill versions. Multiple python installations
(versions) can also be an issue at time.
> - How hard is it to write a test that passes, even if you are just
> running a single test locally on your box. (Running a test on its own
> tends to fail a lot less; interaction with other tests in the same
> directory can complicate things...)
Not an issue for me, though I try to make sure all my tests don't rely
on other tests.
> - How hard is it to debug a test?
gdb is almost impossible. Although the recent changes to mozmill
timeouts make it easier, it still requires extra sleeps (for reference,
I did just about manage it the other day).
The other way of dumps etc, just requires restarting the tests lots.
Both of which, I'm accustomed to.
> - How annoying is the length of time the tests take to run?
folder-display (?) are the most annoying because they take longest. I
generally only run the tests I'm working on though, and use try server
for complete tests if I feel necessary.
> 3) What would make it easier, with degree of preference:
> - Just extracting web-viewable (non-sourcey) docs without further
> categorization from what we have in the helpers.
> - The docs from above, but grouped by what it is they do.
> - Tutorial-style docs.
I think we need some docs to point at Thunderbird's style of MozMill
tests/functions. Then maybe extracted docs from the files.
> 4) Any other thoughts about the mozmill test development setup.
I'm not sure if we should investigate the virtualenv things that Firefox
has implemented. It would take some of the versioning issue away, though
I'm not really convinced about the method they use. Maybe just detecting
versions in runtest.py may be enough.
More information about the tb-planning