External dependent tests in gecko and gaia
edwong at mozilla.com
Wed Aug 13 11:59:23 PDT 2014
A few had discussed manifest vs directory separation of tests. I think the directory idea took root because it’s easily identifiable from it’s path, reducing collision with existing tests. I also agree, using directory as a form of meta data is limiting. That said, both concepts can coexist. We should definitely pursue using manifests to define these tests for execution; that can be next.
On Aug 13, 2014, at 9:39 AM, Gregory Szorc <gps at mozilla.com> wrote:
> On 8/13/14 9:21 AM, Edwin Wong wrote:
>> Hi dev-platform,
>> TL;DR - Cloud Services and Quality Engineering would like to propose the creation of a directory named “external" in gecko and gaia repos for externally dependent tests.
>> This enables features married to Cloud Services such as Loop, FindMyDevice, FirefoxAccounts, and Sync to have centralized tests that can be run locally or on other continuous integration systems. These tests would live in this "external” directory along side existing tests (so they live together). These will be run and sheriffed independently from the main tests. Reviews would be governed by modules and feature teams.
>> More detail:
> I'm glad we're going down this road!
> While having a separate directory enforces the separation between tests and jobs, this separation is partially the result of historical necessity: some test suites looked at filenames or directory contents for determining what to execute.
> These days, tests are defined in manifest files. This means that separate directories don't technically need to exist and the barriers between test flavors can be less pronounced.
> Have you considered annotating external-accessing tests inside manifests and having the automation run the subset of tests it is capable of? e.g. |mach marionette-test --no-external| or |mach test --external-only|
More information about the Dev-fxacct