<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Over the holidays, we noticed that leaks in mochitest and reftest
were not turning jobs orange, and that the test harnesses had been
running in that state for quite some time. During this time several
leak related test failures have landed, which can be tracked with
this dependency tree:<br>
<div class="moz-forward-container"> <a moz-do-not-send="true"
href="https://bugzilla.mozilla.org/showdependencytree.cgi?id=1325148&hide_resolved=0">https://bugzilla.mozilla.org/showdependencytree.cgi?id=1325148&hide_resolved=0</a><br>
<br>
The issue causing jobs to remain green has been fixed, however the
known leak regressions had to be whitelisted to allow this fix to
land. So while future leak regressions will properly fail, the
existing ones (in the dependency tree) still need to be fixed. For
mochitest, the whitelist can be found here:<br>
<a moz-do-not-send="true"
href="https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#2218">https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#2218</a><br>
<br>
Other than that, leak checking is only disabled on linux
crashtests.<br>
<br>
Please take a quick look to see if there is a leak in a component
for which you could help out. I will continue to help with triage
and bisection for the remaining issues until they are all fixed.
Also big thanks to all the people who are currently working on a
fix or have already landed a fix.<br>
<br>
Read on only if you are interested in the details.<br>
<br>
<br>
<u>Why wasn't this caught earlier?<br>
</u><br>
The short answer to this question is that we do not have adequate
testing of our CI.<br>
<u><br>
</u>The problem happened at the intersection between mozharness
and the test harnesses. Basically a change in mozharness exposed a
latent bug in the test harnesses, and was able to land because it
appeared as if nothing went wrong. Catching errors like this is
tricky because regular unit tests would not have detected it
either. It requires integration tests of the CI system as a whole
(spanning test harnesses, mozharness and buildbot/taskcluster).<br>
<br>
<br>
<u>How will we prevent this in the future?</u><br>
<br>
Historically, integration testing our test harnesses has been a
hard problem. However with recent work in taskcluster, python
tests and some refactoring on the build frontend, I believe there
is a path forward that will allow us to stand up this kind of
test. I will commit some of my time to fix this and hope to have <i>something</i>
running that would have caught this by the end of Q1.<br>
<br>
I would also like to stand up a test harness designed to test
command line applications in CI, which would provide another
avenue for writing test harness unit and integration tests. <a
moz-do-not-send="true"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1311991">Bug
1311991</a> will track this work.<br>
<br>
It is important that developers are able to trust our tests, and
when bugs like this happen, that trust is eroded. For that I'd
like to apologize, and express my hope that this will be the last
time a major test result bug like this happens again. At the very
least, we need to have the capability of adding a regression test
when a bug like this happens in the future.<br>
<br>
Thanks for your help and understanding.<br>
- Andrew<br>
</div>
</body>
</html>