quasi-survey on writing mozmill tests for thunderbird

Andrew Sutherland asutherland at asutherland.org
Thu Mar 17 07:35:18 UTC 2011


I'm interested in getting some feedback about the current pain level and 
methodology people are using to write thunderbird mozmill tests.  (This 
is to improve things as opposed to tricking people into outing 
themselves for not having read every file in our large code-base :)

Specifically:

1) Knowing what helper methods to use.  Do people:
- look at existing tests and use the helper methods they find in those?  
(Possibly via copying an existing test and starting from there.)
- look at the code and the jsdoc-style documentation found in the files 
in the shared-modules directory, such as test-folder-display-helpers and 
test-window-helpers.js?
- find that the only official-ish documentation on mozmill tests is for 
mozmill proper and not the thunderbird helpers, and so end up using the 
core mozmill helpers, at least until review time at which they are 
pointed at the wonderful world of test-folder-display-helpers.js and 
friends?

2) How frustrating is mozmill test development?  Specifically:
- How hard is it to get the environment set up?
- How hard is it to write a test that passes, even if you are just 
running a single test locally on your box.  (Running a test on its own 
tends to fail a lot less; interaction with other tests in the same 
directory can complicate things...)
- How hard is it to debug a test?
- How annoying is the length of time the tests take to run?

3) What would make it easier, with degree of preference:
- Just extracting web-viewable (non-sourcey) docs without further 
categorization from what we have in the helpers.
- The docs from above, but grouped by what it is they do.
- Tutorial-style docs.
- Ability to better understand what is happening in local failures with 
logging.  (Noting that we now have two options for this, but they're not 
particularly documented and require varying amounts of setup, both of 
which could potentially be somewhat mitigated.)
- Ability to develop mozmill tests by recording interactions with 
Thunderbird.  (Note: this is actually a hard one, but it's worth knowing 
how desirable it is.)
- other stuff

4) Any other thoughts about the mozmill test development setup.

Andrew



More information about the tb-planning mailing list