c-c external tests (like External dependent tests in gecko and gaia)
asutherland at asutherland.org
Wed Aug 13 20:00:05 UTC 2014
On 08/13/2014 03:00 PM, Kent James wrote:
> I've often wondered if it would not make sense to have testing
> available to external accounts within comm-central, as is discussed
> below in planning for Cloud Services. We could have test accounts on
> various email providers, and do automatic testing of a test suite
> against real email servers. That's how I do my testing on ExQuilla
> using xpcshell tests.
> This must have been discussed at some point in the past, and rejected.
> Does anyone now the history of this?
From the TB 3.0+ era, I think we were already struggling to add
deterministic local test coverage so the external account testing seemed
way out of reach, especially given the credential storage issue and the
potential for the tests to be even more flakey than the ones we
theoretically had absolute control over.
For the Gaia email app, our thoughts have been that testing against real
accounts is something that primarily makes sense:
1) As part of the development/review/test process for changes to the
code. The Gaia email tests actually originally only ran against dovecot
and other real servers. But now we use fake-servers. This is less of a
hassle for people to setup, they run fast because they're local, and we
have complete and absolute control over what they do. It's relatively
easy for a fake-server to exactly emulate the characteristics of some
2) As part of periodic compatibility tests (ex: a few times a day) that
ensure that a server implementation has not changed in a way that breaks
things. This ideally wants to be tracked in a way that it only bothers
you after a sustained set of failures. It's not particularly helpful to
have the tree closed because hotmail.com/outlook.com are down.
The Gaia email app actually has some real-world tests like this that are
run as part of daily smoke-tests but not check-in tests. (Although I
might be confused) For example:
In practice I don't think these have ever actually found anything
useful, just the one hotmail.com outage. But they are also just
super-simple smoke-tests. The test creates an email with a unique
subject, sends it to itself, then verifies it shows up.
There have been some hassles related to logistical issues of running a
bunch of test devices against real-world accounts and all the connection
limit/suspicious activity things that can happen. Also, ActiveSync does
not handle users apparently using a brand new phone/device-id tens of
times a day (https://bugzil.la/1028192).
More information about the tb-planning