External dependent tests in gecko and gaia

Gregory Szorc gps at mozilla.com
Wed Aug 13 09:39:50 PDT 2014

On 8/13/14 9:21 AM, Edwin Wong wrote:
> Hi dev-platform,
> TL;DR - Cloud Services and Quality Engineering would like to propose the creation of a directory named “external" in gecko and gaia repos for externally dependent tests.
> This enables features married to Cloud Services such as Loop, FindMyDevice, FirefoxAccounts, and Sync to have centralized tests that can be run locally or on other continuous integration systems. These tests would live in this "external” directory along side existing tests (so they live together). These will be run and sheriffed independently from the main tests.  Reviews would be governed by modules and feature teams.
> More detail:
> https://wiki.mozilla.org/QA/External_Tests

I'm glad we're going down this road!

While having a separate directory enforces the separation between tests 
and jobs, this separation is partially the result of historical 
necessity: some test suites looked at filenames or directory contents 
for determining what to execute.

These days, tests are defined in manifest files. This means that 
separate directories don't technically need to exist and the barriers 
between test flavors can be less pronounced.

Have you considered annotating external-accessing tests inside manifests 
and having the automation run the subset of tests it is capable of? e.g. 
|mach marionette-test --no-external| or |mach test --external-only|

More information about the Dev-fxacct mailing list