changes to test setup
robert at ocallahan.org
Fri Feb 17 01:03:21 UTC 2017
To make interrmittent test failures easier to figure out, I've made
some significant changes to the test system.
I reorganized the parameters passed to test scripts. Now you can run
an individual test just by running
bash <path>/basic_test.run <test_name>
bash <path>/basic_test.run <test_name>_32
bash <path>/<test>.run <test>_32
from anywhere, if your object directory is accessible from the rr
repository root via ../obj.
I moved the handling of timeouts into a new wrapper command called
test-monitor. The pexpect timeout is set to 10000 so it's basically
irrelevant, and the CTest timeout is set to 1000 seconds in case
test-monitor itself hangs for some reason. test-monitor runs a
subprocess (rr recording, or rr replay possibly under pexpect) with a
timeout specified on the command line. This timeout defaults to 120s
(or 480s on Xeon Phi) but can be overriden by setting `TIMEOUT` in a
test .run script. You no longer configure timeouts for individual
tests in CMakeList.txt.
test-monitor can also be notified directly of an rr failure by sending
a SIGURG signal to it. Its pid is passed in an environment variable
RUNNING_UNDER_TEST_MONITOR. I modified rr so that uses of
emergency_debug() and abort() (now notify_abort() in util.cc) send
SIGURG to test-monitor if present. I haven't modified uses of assert()
but that might be good to do.
When test-monitor detects a timeout or SIGURG notification, it gathers
a pile of data and prints it to a file passed on its command line.
util.sh directs that output to record.err, replay.err or a new
debug.err, which then get printed as a failing test exits. I've also
modified util.sh to print gdb_rr.log when a debugger test fails.
The gathered data is:
-- /proc/<pid>/status and /proc/<pid>/stack for every process and
thread in the process subtree rooted at test-monitor's child.
-- test-monitor tries to attach gdb to the rr process and get stack
traces of all rr threads.
-- If rr triggered emergency_debug(), test-monitor tries to attach to
the emergency debugger and dump the registers and stack trace of the
The last step is a bit risky because it could cause test-monitor to
hang if rr is in a particularly bad state. I hope this won't be much
of a problem in practice. If test-monitor hangs and is timed out by
CTest, then its output up to that point will still be in some .err
file in the rr-test temp directory, so the output won't show up in the
CTest output but is still accessible if you have filesystem access.
lbir ye,ea yer.tnietoehr rdn rdsme,anea lurpr edna e hnysnenh hhe uresyf toD
selthor stor edna siewaoeodm or v sstvr esBa kbvted,t rdsme,aoreseoouoto
o l euetiuruewFa kbn e hnystoivateweh uresyf tulsa rehr rdm or rnea lurpr
.a war hsrer holsa rodvted,t nenh hneireseoouot.tniesiewaoeivatewt sstvr esn
More information about the rr-dev