c-c external tests (like External dependent tests in gecko and gaia)

Andrew Sutherland asutherland at asutherland.org
Wed Aug 13 20:00:05 UTC 2014


On 08/13/2014 03:00 PM, Kent James wrote:
> I've often wondered if it would not make sense to have testing 
> available to external accounts within comm-central, as is discussed 
> below in planning for Cloud Services. We could have test accounts on 
> various email providers, and do automatic testing of a test suite 
> against real email servers. That's how I do my testing on ExQuilla 
> using xpcshell tests.
>
> This must have been discussed at some point in the past, and rejected. 
> Does anyone now the history of this?

 From the TB 3.0+ era, I think we were already struggling to add 
deterministic local test coverage so the external account testing seemed 
way out of reach, especially given the credential storage issue and the 
potential for the tests to be even more flakey than the ones we 
theoretically had absolute control over.


For the Gaia email app, our thoughts have been that testing against real 
accounts is something that primarily makes sense:

1) As part of the development/review/test process for changes to the 
code.  The Gaia email tests actually originally only ran against dovecot 
and other real servers.  But now we use fake-servers.  This is less of a 
hassle for people to setup, they run fast because they're local, and we 
have complete and absolute control over what they do.  It's relatively 
easy for a fake-server to exactly emulate the characteristics of some 
other server.

2) As part of periodic compatibility tests (ex: a few times a day) that 
ensure that a server implementation has not changed in a way that breaks 
things.  This ideally wants to be tracked in a way that it only bothers 
you after a sustained set of failures.  It's not particularly helpful to 
have the tree closed because hotmail.com/outlook.com are down.


The Gaia email app actually has some real-world tests like this that are 
run as part of daily smoke-tests but not check-in tests. (Although I 
might be confused)  For example:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/email/test_setup_and_send_imap_email.py

In practice I don't think these have ever actually found anything 
useful, just the one hotmail.com outage.  But they are also just 
super-simple smoke-tests.  The test creates an email with a unique 
subject, sends it to itself, then verifies it shows up.

There have been some hassles related to logistical issues of running a 
bunch of test devices against real-world accounts and all the connection 
limit/suspicious activity things that can happen.  Also, ActiveSync does 
not handle users apparently using a brand new phone/device-id tens of 
times a day (https://bugzil.la/1028192).


Andrew



More information about the tb-planning mailing list