Proposal: Removing make targets for running tests

Gregory Szorc gps at mozilla.com
Fri Aug 30 20:59:26 UTC 2013


tl;dr the tree currently allows you to run tests via make targets and 
mach. The mach commands are more intuitive and easier to support going 
forward, so I'm proposing we remove support for using make targets to 
run tests. Please take the survey at 
http://www.surveymonkey.com/s/H5LB2GF to respond without clogging inboxes.

Long version.

The in-tree mechanism for running tests has always been a bit fragile. 
It uses separate code paths from what is executed in release automation 
(for a few reasons which I won't explain here).

For years, people have used make targets to run tests. They get the job 
done, sure. But they are limited in what you can do and are awkward to 
use e.g. specifying arguments or options to influence run-time behavior 
involves setting environment variables (that's how make works). These 
make targets are effectively shell scripts and are an abuse of make as a 
dependency evaluation tool. As a build system maintainer, I don't want 
these make targets polluting the core build system.

mach, by contrast, is easier to use. You can type |mach help 
xpcshell-test| and get friendly output [1] telling you how to influence 
test running. Even better - these docs are auto-generated from what's in 
the tree, so MDN is never out of sync. It's easier to add features to 
mach commands than make (Python is a better systems language than 
shell/make). And, mach - being written in Python, the same language as 
the test harnesses - integrates much more easily into the test 
harnesses. You can perform deep, native integration into the test 
harnesses. This facilitates interesting future possibilities, such as 
mach commands that simultaneously invoke multiple test harnesses and 
interleave their results - something not possible with just make. I've 
even seen someone run an HTTP+WebSocket server from a mach command and 
pipe test results to a web browser in real-time! Oh the possibilities! 
Long term, the mach command for running tests and the code path for 
running tests on release automation can converge. This will never happen 
with make targets.

mach is the future for running tests. The make targets are effectively 
deprecated and have become more burdensome to maintain. They have broken 
many times in the last few months because people changing the test 
harnesses are all using mach and didn't test the make targets. They are 
baggage preserved only so developers aren't forced to change. That 
baggage has become too costly to maintain. I propose we eliminate the 
make targets for running tests.

Aside from obvious "I don't want to change" complaints, are there any 
objections to this change? Is anyone not able to run the mach testing 
commands today? Are there features missing from the mach commands? 
Please file bugs!

Please take the survey at [2] lend your voice to this decision.

[1] https://gps.pastebin.mozilla.org/2942348
[2] http://www.surveymonkey.com/s/H5LB2GF



More information about the firefox-dev mailing list