changes to test setup

Robert O'Callahan robert at
Fri Feb 17 01:03:21 UTC 2017

To make interrmittent test failures easier to figure out, I've made
some significant changes to the test system.

I reorganized the parameters passed to test scripts. Now you can run
an individual test just by running
      bash <path>/ <test_name>
      bash <path>/ <test_name>_32
      bash <path>/<test>.run
      bash <path>/<test>.run <test>_32
from anywhere, if your object directory is accessible from the rr
repository root via ../obj.

I moved the handling of timeouts into a new wrapper command called
test-monitor. The pexpect timeout is set to 10000 so it's basically
irrelevant, and the CTest timeout is set to 1000 seconds in case
test-monitor itself hangs for some reason. test-monitor runs a
subprocess (rr recording, or rr replay possibly under pexpect) with a
timeout specified on the command line. This timeout defaults to 120s
(or 480s on Xeon Phi) but can be overriden by setting `TIMEOUT` in a
test .run script. You no longer configure timeouts for individual
tests in CMakeList.txt.

test-monitor can also be notified directly of an rr failure by sending
a SIGURG signal to it. Its pid is passed in an environment variable
RUNNING_UNDER_TEST_MONITOR. I modified rr so that uses of
emergency_debug() and abort() (now notify_abort() in send
SIGURG to test-monitor if present. I haven't modified uses of assert()
but that might be good to do.

When test-monitor detects a timeout or SIGURG notification, it gathers
a pile of data and prints it to a file passed on its command line. directs that output to record.err, replay.err or a new
debug.err, which then get printed as a failing test exits. I've also
modified to print gdb_rr.log when a debugger test fails.

The gathered data is:
-- /proc/<pid>/status and /proc/<pid>/stack for every process and
thread in the process subtree rooted at test-monitor's child.
-- test-monitor tries to attach gdb to the rr process and get stack
traces of all rr threads.
-- If rr triggered emergency_debug(), test-monitor tries to attach to
the emergency debugger and dump the registers and stack trace of the
The last step is a bit risky because it could cause test-monitor to
hang if rr is in a particularly bad state. I hope this won't be much
of a problem in practice. If test-monitor hangs and is timed out by
CTest, then its output up to that point will still be in some .err
file in the rr-test temp directory, so the output won't show up in the
CTest output but is still accessible if you have filesystem access.

lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr  esn

More information about the rr-dev mailing list