quasi-survey on writing mozmill tests for thunderbird
asutherland at asutherland.org
Thu Mar 17 07:35:18 UTC 2011
I'm interested in getting some feedback about the current pain level and
methodology people are using to write thunderbird mozmill tests. (This
is to improve things as opposed to tricking people into outing
themselves for not having read every file in our large code-base :)
1) Knowing what helper methods to use. Do people:
- look at existing tests and use the helper methods they find in those?
(Possibly via copying an existing test and starting from there.)
- look at the code and the jsdoc-style documentation found in the files
in the shared-modules directory, such as test-folder-display-helpers and
- find that the only official-ish documentation on mozmill tests is for
mozmill proper and not the thunderbird helpers, and so end up using the
core mozmill helpers, at least until review time at which they are
pointed at the wonderful world of test-folder-display-helpers.js and
2) How frustrating is mozmill test development? Specifically:
- How hard is it to get the environment set up?
- How hard is it to write a test that passes, even if you are just
running a single test locally on your box. (Running a test on its own
tends to fail a lot less; interaction with other tests in the same
directory can complicate things...)
- How hard is it to debug a test?
- How annoying is the length of time the tests take to run?
3) What would make it easier, with degree of preference:
- Just extracting web-viewable (non-sourcey) docs without further
categorization from what we have in the helpers.
- The docs from above, but grouped by what it is they do.
- Tutorial-style docs.
- Ability to better understand what is happening in local failures with
logging. (Noting that we now have two options for this, but they're not
particularly documented and require varying amounts of setup, both of
which could potentially be somewhat mitigated.)
- Ability to develop mozmill tests by recording interactions with
Thunderbird. (Note: this is actually a hard one, but it's worth knowing
how desirable it is.)
- other stuff
4) Any other thoughts about the mozmill test development setup.
More information about the tb-planning