<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>A couple of sneak peeks from the next update:</p>
<p>- We have daily reports on <a class="moz-txt-link-freetext"
href="https://coveralls.io/github/marco-c/gecko-dev">https://coveralls.io/github/marco-c/gecko-dev</a>
and <a class="moz-txt-link-freetext"
href="https://codecov.io/gh/marco-c/gecko-dev">https://codecov.io/gh/marco-c/gecko-dev</a>
(tests with e10s on are currently disabled, but should be enabled
soon; we are currently investigating the differences between e10s
and non-e10s tests).<br>
</p>
<p>- There is now a simple way to spin a try code coverage build and
generate a report from a one-click loaner, see
<a class="moz-txt-link-freetext"
href="https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox#Generate_Code_Coverage_report_from_a_try_build_%28or_any_other_treeherder_build%29">https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build)</a>.</p>
<p>- Marco.<br>
</p>
<br>
<div class="moz-cite-prefix">Il 28/04/17 04:39, Kyle Lahnakoski ha
scritto:<br>
</div>
<blockquote
cite="mid:%3Ccfe584b6-0d9a-af49-06b4-3675fb28c00d@mozilla.com%3E"
type="cite">
<meta http-equiv="content-type" content="text/html;
charset=windows-1252">
<p><br>
</p>
<div class="moz-forward-container">Good day Firefox Dev!<br>
<br>
We have a Code Coverage project, and we have come quite far
Would you like to help? We have a meeting <span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z">
every Friday @ 11:30 EDT (08:30 PDT)</span> <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://public.etherpad-mozilla.org/p/code_coverage_Q1_17">https://public.etherpad-mozilla.org/p/code_coverage_Q1_17</a><br>
<br>
Highlights:<br>
<br>
* In the past month, mcastelluccio in Release Management has
built grcov to speed artifact processing, he has also been
pushing coverage to <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://coveralls.io/github/marco-c/gecko-dev">https://coveralls.io/github/marco-c/gecko-dev</a><br>
* npark and irios are generating Android and iOS coverage
respectivly and porting to <span
class="author-a-z78zgz71zz85zz78zz88zz79zz82zz87zz74zz78z51z72zz83z1
url"><a moz-do-not-send="true"
href="https://codecov.io/gh/mozilla-mobile/firefox-ios/commit/d1adc1f87c31dc9343a540ffcae94b769eb4556d/graphs">https://codecov.io/gh/mozilla-mobile/firefox-ios</a></span><br>
* We are running JS and C++ code coverage nightly on Taskcluster
for a variety of test suites.<br>
* A number of students had been working on the code coverage ETL
pipeline, and unified UI, for course credit<br>
* The ETL pipeline feeds a central datastore of all coverage,
and the UI queries it for presentation<br>
<br>
Here is April's update (Wed, 5 Apr 2017 ). I will be sending
out May's next week.<br>
<br>
<div class="moz-forward-container"><span style="color: #b22222;
">## The Progress to Date</span><br>
<br>
<span style="color: #808080; ">* </span>Coverage is scheduled
to run daily on central. <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=cov">https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=cov</a><br>
<span style="color: #808080; ">* </span>The UCOSP[1] students
have completed their term, and can show aggregate statistics
by folder with the code coverage ui [5] <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://ericdesj.github.io/moz-coco-w17-preview/">https://ericdesj.github.io/moz-coco-w17-preview/</a>
<br>
* <span style="color: #808080; "></span>c/c++ coverage is
being ETL'ed into our unified coverage datastore, but does not
yet show in the UI<br>
<span style="color: #808080; ">* </span><span style="color:
#008000; ">`grcov`</span>[4] was written mcastelluccio to
process coverage artifacts much faster; reducing processing
time from hours to seconds. <span style="color: #008000; ">`grcov`</span> also
outputs multiple formats, including for coveralls.io<br>
<span style="color: #808080; ">* </span>There is exploration
into using coveralls.io: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://coveralls.io/jobs/24085087">https://coveralls.io/jobs/24085087</a>,
the main page is <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://coveralls.io/github/marco-c/gecko-dev">https://coveralls.io/github/marco-c/gecko-dev</a>.<br>
* JSVM coverage got working today! <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce2c56eeceb1bc252c6b25a50304f725c320d5e0">https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce2c56eeceb1bc252c6b25a50304f725c320d5e0</a>
<br>
<br>
<span style="color: #b22222; ">## The Blockers</span><br>
<br>
These items are further behind than I would like. If you can
help, please do.<br>
<br>
<span style="color: #808080; ">* </span>Getting Rust to emit
coverage artifacts is important: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1335518">https://bugzilla.mozilla.org/show_bug.cgi?id=1335518</a><br>
<span style="color: #808080; ">* </span>Continuing the hard
work of enabling C/C++ codecoverage on all our tests: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1301170">https://bugzilla.mozilla.org/show_bug.cgi?id=1301170</a><br>
<span style="color: #808080; "></span><br>
<span style="color: #b22222; ">## The Biggest Problem</span><br>
<br>
<span style="font-weight: bold; ">**Our test suites are
unstable**</span>. Many of you may already know that many
tests fail intermittently; these "intermittents" impact code
coverage; either by breaking the test run, and preventing
coverage collection; or by changing code paths. The challenge
is automating the metatdata collection and having a strategy
to mitigate the loss of data. [8] <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1337241">https://bugzilla.mozilla.org/show_bug.cgi?id=1337241</a><br>
<br>
Even so, <span style="font-weight: bold; ">**test runs are not
deterministic**</span>, so we expect fluctuation in the
aggregate coverage statistics; How large this fluctuation is
we do not know.<br>
<br>
Got ideas? You want to work on a problem that no one will
thank you for? ;) Please talk to me! [3]<br>
<br>
<span style="color: #b22222; ">## The Current Plan</span><br>
<br>
For the next month we will be continuing work on:<br>
<br>
<span style="color: #808080; ">* </span>validating the data
and process, including adding Rust and more test suites. <br>
* <span style="color: #808080; "></span>Adding per-line
coverage to the CoCo UI: The aggregate numbers are good for
identifying sections of code that are not tested, but we need
line-level details to inform action. <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://github.com/mozilla/moz-coco/issues/7">https://github.com/mozilla/moz-coco/issues/7</a><br>
* Explore how coveralls.io can be used to make that job
easier <br>
<span style="color: #808080; ">* </span>Explore how to
leverage Taskcluster-generated coverage artifacts with local
coverage tools [9] <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1350446">https://bugzilla.mozilla.org/show_bug.cgi?id=1350446</a><br>
* We are investigating if coverage tools behave differently
when e10s is enabled, then enabling it.<br>
* JSVM ETL pipeline work will commence: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1301174">https://bugzilla.mozilla.org/show_bug.cgi?id=1301174</a><br>
<br>
<br>
<span style="color: #b22222; ">## Motivation</span><br>
<br>
<span style="font-style: italic; ">*Unchanged from last email*</span> -
Knowing code coverage statistics and code coverage specifics
can help evaluate risk in code: It can show what test suites
cover what lines, it can show if new code is covered by an
existing test. There are many other exciting insights we can
extract from code coverage. But first, we must collect it.<br>
<br>
<span style="color: #b22222; ">## Reference</span><br>
<br>
[1] UCOSP (Undergraduate Capstone Open Source Projects) <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://ucosp.ca/">http://ucosp.ca/</a><br>
<br>
[2] The original plan: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#">https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#</a><br>
<br>
[3] Kyle Lahnakoski email: <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:klahnakoski@mozilla.org">klahnakoski@mozilla.org</a> irc:
ekyle on #ateam@irc.mozilla.org<br>
<br>
[4] grcov <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://github.com/marco-c/grcov">https://github.com/marco-c/grcov</a><br>
<br>
[5] (Co)de (Co)verage UI preview - <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://ericdesj.github.io/moz-coco-w17-preview/">https://ericdesj.github.io/moz-coco-w17-preview/</a><br>
<br>
[5b] CoCo is on Github - <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://github.com/mozilla/moz-coco">https://github.com/mozilla/moz-coco</a><br>
<br>
[6] CoCo relay - <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://github.com/ericdesj/moz-coco-relay">https://github.com/ericdesj/moz-coco-relay</a><br>
<br>
[7] Rust Coverage Tracking <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1335518">https://bugzilla.mozilla.org/show_bug.cgi?id=1335518</a><br>
<br>
[8] Knowing when the coverage is complete: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1337241">https://bugzilla.mozilla.org/show_bug.cgi?id=1337241</a><br>
<br>
[9] Coverage with local tools: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1350446">https://bugzilla.mozilla.org/show_bug.cgi?id=1350446</a>
</div>
<br>
<br>
<br>
<br>
</div>
</blockquote>
<br>
</body>
</html>