improved test reliability

Chris Jones jones.chris.g at gmail.com
Sun Jul 6 20:59:44 PDT 2014


I finished three runs and saw intermittent failures on 2 tests (two
separate runs).  One of them I've seen before, one I haven't.  They seem
related though.  Have to run off, will file and try to debug when I have
time.

Out of curiosity, which kernel/microarch are you testing on?  I'm surprised
we're seeing such different results.  (I'm on 3.14.2-200.fc20.x86_64 / Ivy
Bridge.)

Also, the ctest harness uses historical test run times to order the tests
for minimum suite run-time.  That's what you want for everyday dev, but for
catching extreme edge case failures it's not so much.  Blowing away your
objdir in between test runs will also destroy the run history and may
inject some more nondeterminism in the test run sequence.

Cheers,
Chris


On Fri, Jul 4, 2014 at 11:35 PM, Robert O'Callahan <robert at ocallahan.org>
wrote:

> Over the last couple of weeks I've been running the rr tests a lot and
> fixing bugs I found in the tests and in rr itself. Last night I ran 100
> iterations of the test suite on my laptop and another 100 in a VM with no
> test failures. So if you do see any failures running the test suite against
> rr master it's definitely worth filing an issue!
>
> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
> le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w
>
> _______________________________________________
> rr-dev mailing list
> rr-dev at mozilla.org
> https://mail.mozilla.org/listinfo/rr-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/rr-dev/attachments/20140706/43d932b4/attachment.html>


More information about the rr-dev mailing list