<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<div class="moz-forward-container">Good day Firefox Dev!<br>
<br>
We have a Code Coverage project, and we have come quite far Would
you like to help? We have a meeting <span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z">
every Friday @ 11:30 EDT (08:30 PDT)</span>
<a class="moz-txt-link-freetext" href="https://public.etherpad-mozilla.org/p/code_coverage_Q1_17">https://public.etherpad-mozilla.org/p/code_coverage_Q1_17</a><br>
<br>
Highlights:<br>
<br>
* In the past month, mcastelluccio in Release Management has built
grcov to speed artifact processing, he has also been pushing
coverage to <a class="moz-txt-link-freetext" href="https://coveralls.io/github/marco-c/gecko-dev">https://coveralls.io/github/marco-c/gecko-dev</a><br>
* npark and irios are generating Android and iOS coverage
respectivly and porting to <span
class="author-a-z78zgz71zz85zz78zz88zz79zz82zz87zz74zz78z51z72zz83z1
url"><a
href="https://codecov.io/gh/mozilla-mobile/firefox-ios/commit/d1adc1f87c31dc9343a540ffcae94b769eb4556d/graphs">https://codecov.io/gh/mozilla-mobile/firefox-ios</a></span><br>
* We are running JS and C++ code coverage nightly on Taskcluster
for a variety of test suites.<br>
* A number of students had been working on the code coverage ETL
pipeline, and unified UI, for course credit<br>
* The ETL pipeline feeds a central datastore of all coverage, and
the UI queries it for presentation<br>
<br>
Here is April's update (Wed, 5 Apr 2017 ). I will be sending out
May's next week.<br>
<br>
<div class="moz-forward-container"><span style="color: #b22222; ">##
The Progress to Date</span><br>
<br>
<span style="color: #808080; ">* </span>Coverage is scheduled to
run daily on central. <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=cov">https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=cov</a><br>
<span style="color: #808080; ">* </span>The UCOSP[1] students
have completed their term, and can show aggregate statistics by
folder with the code coverage ui [5] <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://ericdesj.github.io/moz-coco-w17-preview/">https://ericdesj.github.io/moz-coco-w17-preview/</a>
<br>
* <span style="color: #808080; "></span>c/c++ coverage is being
ETL'ed into our unified coverage datastore, but does not yet
show in the UI<br>
<span style="color: #808080; ">* </span><span style="color:
#008000; ">`grcov`</span>[4] was written mcastelluccio to
process coverage artifacts much faster; reducing processing time
from hours to seconds. <span style="color: #008000; ">`grcov`</span> also
outputs multiple formats, including for coveralls.io<br>
<span style="color: #808080; ">* </span>There is exploration
into using coveralls.io: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://coveralls.io/jobs/24085087">https://coveralls.io/jobs/24085087</a>,
the main page is <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://coveralls.io/github/marco-c/gecko-dev">https://coveralls.io/github/marco-c/gecko-dev</a>.<br>
* JSVM coverage got working today!
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce2c56eeceb1bc252c6b25a50304f725c320d5e0">https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce2c56eeceb1bc252c6b25a50304f725c320d5e0</a>
<br>
<br>
<span style="color: #b22222; ">## The Blockers</span><br>
<br>
These items are further behind than I would like. If you can
help, please do.<br>
<br>
<span style="color: #808080; ">* </span>Getting Rust to emit
coverage artifacts is important: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1335518">https://bugzilla.mozilla.org/show_bug.cgi?id=1335518</a><br>
<span style="color: #808080; ">* </span>Continuing the hard
work of enabling C/C++ codecoverage on all our tests: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1301170">https://bugzilla.mozilla.org/show_bug.cgi?id=1301170</a><br>
<span style="color: #808080; "></span><br>
<span style="color: #b22222; ">## The Biggest Problem</span><br>
<br>
<span style="font-weight: bold; ">**Our test suites are
unstable**</span>. Many of you may already know that many
tests fail intermittently; these "intermittents" impact code
coverage; either by breaking the test run, and preventing
coverage collection; or by changing code paths. The challenge is
automating the metatdata collection and having a strategy to
mitigate the loss of data. [8] <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1337241">https://bugzilla.mozilla.org/show_bug.cgi?id=1337241</a><br>
<br>
Even so, <span style="font-weight: bold; ">**test runs are not
deterministic**</span>, so we expect fluctuation in the
aggregate coverage statistics; How large this fluctuation is we
do not know.<br>
<br>
Got ideas? You want to work on a problem that no one will thank
you for? ;) Please talk to me! [3]<br>
<br>
<span style="color: #b22222; ">## The Current Plan</span><br>
<br>
For the next month we will be continuing work on:<br>
<br>
<span style="color: #808080; ">* </span>validating the data and
process, including adding Rust and more test suites. <br>
* <span style="color: #808080; "></span>Adding per-line
coverage to the CoCo UI: The aggregate numbers are good for
identifying sections of code that are not tested, but we need
line-level details to inform action. <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://github.com/mozilla/moz-coco/issues/7">https://github.com/mozilla/moz-coco/issues/7</a><br>
* Explore how coveralls.io can be used to make that job easier <br>
<span style="color: #808080; ">* </span>Explore how to leverage
Taskcluster-generated coverage artifacts with local coverage
tools [9] <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1350446">https://bugzilla.mozilla.org/show_bug.cgi?id=1350446</a><br>
* We are investigating if coverage tools behave differently when
e10s is enabled, then enabling it.<br>
* JSVM ETL pipeline work will commence: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1301174">https://bugzilla.mozilla.org/show_bug.cgi?id=1301174</a><br>
<br>
<br>
<span style="color: #b22222; ">## Motivation</span><br>
<br>
<span style="font-style: italic; ">*Unchanged from last email*</span> -
Knowing code coverage statistics and code coverage specifics can
help evaluate risk in code: It can show what test suites cover
what lines, it can show if new code is covered by an existing
test. There are many other exciting insights we can extract
from code coverage. But first, we must collect it.<br>
<br>
<span style="color: #b22222; ">## Reference</span><br>
<br>
[1] UCOSP (Undergraduate Capstone Open Source Projects) <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://ucosp.ca/">http://ucosp.ca/</a><br>
<br>
[2] The original plan: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#">https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#</a><br>
<br>
[3] Kyle Lahnakoski email: <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:klahnakoski@mozilla.org">klahnakoski@mozilla.org</a> irc:
ekyle on #ateam@irc.mozilla.org<br>
<br>
[4] grcov <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://github.com/marco-c/grcov">https://github.com/marco-c/grcov</a><br>
<br>
[5] (Co)de (Co)verage UI preview - <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://ericdesj.github.io/moz-coco-w17-preview/">https://ericdesj.github.io/moz-coco-w17-preview/</a><br>
<br>
[5b] CoCo is on Github - <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://github.com/mozilla/moz-coco">https://github.com/mozilla/moz-coco</a><br>
<br>
[6] CoCo relay - <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://github.com/ericdesj/moz-coco-relay">https://github.com/ericdesj/moz-coco-relay</a><br>
<br>
[7] Rust Coverage Tracking <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1335518">https://bugzilla.mozilla.org/show_bug.cgi?id=1335518</a><br>
<br>
[8] Knowing when the coverage is complete: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1337241">https://bugzilla.mozilla.org/show_bug.cgi?id=1337241</a><br>
<br>
[9] Coverage with local tools: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1350446">https://bugzilla.mozilla.org/show_bug.cgi?id=1350446</a>
</div>
<br>
<br>
<br>
<br>
</div>
</body>
</html>