c-c external tests (like External dependent tests in gecko and gaia)

Joshua Cranmer 🐧 Pidgeot18 at gmail.com
Wed Aug 13 20:09:22 UTC 2014


On 8/13/2014 2:00 PM, Kent James wrote:
> I've often wondered if it would not make sense to have testing 
> available to external accounts within comm-central, as is discussed 
> below in planning for Cloud Services. We could have test accounts on 
> various email providers, and do automatic testing of a test suite 
> against real email servers. That's how I do my testing on ExQuilla 
> using xpcshell tests.
>
> This must have been discussed at some point in the past, and rejected. 
> Does anyone now the history of this?

Actually, it would be very useful to have tests against real servers, 
particularly for extremely hard-to-mock things like GSSAPI or LDAP. The 
problem with testing against real email servers in particular are the 
following:
1. Uptime of accounts (not so much a problem for the largest webmail 
providers)
2. Concurrency issues (two developers happen to run their tests at the 
same time--how do you deal with that?)
3. Reproducibility issues. If email providers do A/B testing, you can 
end up with tests that fail for some developers but not others.
4. Account limits--automated tests that run relatively frequently can 
end up accidentally triggering lockouts.

If Mozilla ran some servers on its networks for our uses (e.g., an 
Exchange server, Dovecot, Cyrus, etc.), and we could give out access to 
these servers for tests and contributors, that would help much of the 
problem. I recall Gaia had plans to switch to real servers for mail 
testing; CC'ing asuth for more information about what Gaia's plans were 
here.

-- 
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist




More information about the tb-planning mailing list