quasi-survey on writing mozmill tests for thunderbird

Jonathan Protzenko jonathan.protzenko at gmail.com
Thu Mar 17 14:51:00 UTC 2011


I'm not the biggest user of Mozmill, but I've written quite a few tests 
since I started submitting patches, so I think I may have some feedback 
to give. Apologies if this too lengthy / confuse.

On 03/17/2011 08:35 AM, Andrew Sutherland wrote:
> 1) Knowing what helper methods to use.  Do people:
> - look at existing tests and use the helper methods they find in
> those?  (Possibly via copying an existing test and starting from there.)
> - look at the code and the jsdoc-style documentation found in the
> files in the shared-modules directory, such as
> test-folder-display-helpers and test-window-helpers.js?
> - find that the only official-ish documentation on mozmill tests is
> for mozmill proper and not the thunderbird helpers, and so end up
> using the core mozmill helpers, at least until review time at which
> they are pointed at the wonderful world of
> test-folder-display-helpers.js and friends?
It took me quite some time to understand that there was basically no 
documentation, save for reading the Thunderbird helpers code. The 
Mozmill pages on MDC (imho) do not state clearly enough that they're 
about the general ElementsLib thing, and that they're no use if you're 
looking for documentation about Thunderbird-specific Mozmill.

After doing that, I usually copy/paste an existing test, import 
test_folder_display_helpers.js, and start doing stuff. The real problem 
is I usually have no clue how to achieve what I'm trying to do. I 
usually go grepping frantically in the mail/test/mozmill folder for 
words that may be related to what I'm trying to do. Not very efficient, 
but that's the only way I've found to learn about existing functions.

The last big test I wrote was about the context menu, in a content tab. 
I first started synthesizing the right-click event myself, because the 
only examples I could find were about elements that were in the usual 
chrome window.

mc.rightClick(mc.eid("someChromeElement"))

Because the element I wanted to right-click was in a content tab, I just 
couldn't use mc.eid. Then, after painful hours of reading the DOM Event 
stuff on MDC, I ended up synthesizing the right-click event myself. 
Shortly after that, I figured out I could trick Mozmill by using:

mc.rightClick({ getNode: function() 
myContentDoc.getElementById("myContentElement") });

Then, after a few more hours, I finally wrote.

mc.rightClick(new elementslib.Elem(myContentElement));

Finding about the close_popup thing took me some time as well. In a 
nutshell, I'd say there's often a way of doing things easily and quickly 
with mozmill, except that it always take me hours finding it.
>
> 2) How frustrating is mozmill test development?  Specifically:
> - How hard is it to get the environment set up?
> - How hard is it to write a test that passes, even if you are just
> running a single test locally on your box.  (Running a test on its own
> tends to fail a lot less; interaction with other tests in the same
> directory can complicate things...)
> - How hard is it to debug a test?
> - How annoying is the length of time the tests take to run?
Getting the environment setup wasn't hard to me, it's just that finding 
the right set of commands to launch a specific test requires plowing 
through MDC until you find the only line that tells you about the 
SOLO_TEST thingy. Plus, since Mozmill now launches a VNC server on 
linux, I had to build a small shell script that launches the test, waits 
three seconds, launches VNC and tells it it's actually local so there's 
no need to prompt me for the password, etc. That's not much, but I had 
to read quite a few man-pages to have a command line that's suitable for 
a quick edit-and-run dev cycle.

Once someone told me about mc.sleep() on IRC, debugging tests got much 
easier. The length of tests is not an issue for me.
>
> 3) What would make it easier, with degree of preference:
> - Just extracting web-viewable (non-sourcey) docs without further
> categorization from what we have in the helpers.
> - The docs from above, but grouped by what it is they do.
> - Tutorial-style docs.
> - Ability to better understand what is happening in local failures
> with logging.  (Noting that we now have two options for this, but
> they're not particularly documented and require varying amounts of
> setup, both of which could potentially be somewhat mitigated.)
> - Ability to develop mozmill tests by recording interactions with
> Thunderbird.  (Note: this is actually a hard one, but it's worth
> knowing how desirable it is.)
> - other stuff
I think a big-picture view of Mozmill, and a simple tutorial that 
teaches you where to read the documentation, how the elements lib 
relates to Thunderbird-specific Mozmill helpers, would greatly help 
newcomers. A few tips and tricks to easily run tests on Linux would be 
valuable as well. That's also the place you could tell people about 
debugging tricks, like mc.sleep(10000), etc.

Second, I'd love to see an official JSDoc thing for Thunderbird. I know 
you've got your own docs for Gloda for instance, on your visophyte 
server, but they're out of date, and only one page on MDC links to them. 
Maybe a central repository for all JSDoc generated from newer 
Thunderbird code would be valuable? That would allow one to quickly get 
an overview of all available functions in Mozmill helpers. If they were 
grouped by "what they do", that'd be fantastic. And maybe we could add 
in there documentation for all the gloda modules, for tabmail.xml, etc. etc.
>
> 4) Any other thoughts about the mozmill test development setup.
It's somehow frustrating that I should always spend more time writing 
the test than writing the patch, but well... I guess there's not much we 
can do about it :-).

Hope that was helpful,

jonathan
>
> Andrew
> _______________________________________________
> tb-planning mailing list
> tb-planning at mozilla.org
> https://mail.mozilla.org/listinfo/tb-planning



More information about the tb-planning mailing list