<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>Thanks for the update. This looks really good!<br></div>
<div><br></div>
<div>Kanru<br></div>
<div><br></div>
<div>On Tue, May 2, 2017, at 06:51 PM, Marco Castelluccio wrote:<br></div>
<blockquote type="cite"><p>A couple of sneak peeks from the next update:<br></p><p>- We have daily reports on <a href="https://coveralls.io/github/marco-c/gecko-dev">https://coveralls.io/github/marco-c/gecko-dev</a> and <a href="https://codecov.io/gh/marco-c/gecko-dev">https://codecov.io/gh/marco-c/gecko-dev</a> (tests with e10s on are currently disabled, but should be enabled
soon; we are currently investigating the differences between e10s
and non-e10s tests).<br></p><p>- There is now a simple way to spin a try code coverage build and
generate a report from a one-click loaner, see <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox#Generate_Code_Coverage_report_from_a_try_build_%28or_any_other_treeherder_build%29">https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build)</a>.<br></p><p>- Marco.<br></p><div><br></div>
<div>Il 28/04/17 04:39, Kyle Lahnakoski ha
scritto:<br></div>
<blockquote type="cite"><p><br></p><div><div>Good day Firefox Dev!<br></div>
<div> <br></div>
<div> We have a Code Coverage project, and we have come quite far
Would you like to help? We have a meeting <span> every Friday @ 11:30 EDT (08:30 PDT)</span> <a href="https://public.etherpad-mozilla.org/p/code_coverage_Q1_17">https://public.etherpad-mozilla.org/p/code_coverage_Q1_17</a><br></div>
<div> <br></div>
<div> Highlights:<br></div>
<div> <br></div>
<div> * In the past month, mcastelluccio in Release Management has
built grcov to speed artifact processing, he has also been
pushing coverage to <a href="https://coveralls.io/github/marco-c/gecko-dev">https://coveralls.io/github/marco-c/gecko-dev</a><br></div>
<div> * npark and irios are generating Android and iOS coverage
respectivly and porting to <span><a href="https://codecov.io/gh/mozilla-mobile/firefox-ios/commit/d1adc1f87c31dc9343a540ffcae94b769eb4556d/graphs">https://codecov.io/gh/mozilla-mobile/firefox-ios</a></span><br></div>
<div> * We are running JS and C++ code coverage nightly on Taskcluster
for a variety of test suites.<br></div>
<div> * A number of students had been working on the code coverage ETL
pipeline, and unified UI, for course credit<br></div>
<div> * The ETL pipeline feeds a central datastore of all coverage,
and the UI queries it for presentation<br></div>
<div> <br></div>
<div> Here is April's update (Wed, 5 Apr 2017 ). I will be sending
out May's next week.<br></div>
<div> <br></div>
<div> <br></div>
<div><div><span class="colour" style="color:rgb(178, 34, 34)">## The Progress to Date</span><br></div>
<div> <br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)">* </span>Coverage is scheduled
to run daily on central. <a href="https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=cov">https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=cov</a><br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)">* </span>The UCOSP[1] students
have completed their term, and can show aggregate statistics
by folder with the code coverage ui [5] <a href="https://ericdesj.github.io/moz-coco-w17-preview/">https://ericdesj.github.io/moz-coco-w17-preview/</a> <br></div>
<div> * <span class="colour" style="color:rgb(128, 128, 128)"></span>c/c++ coverage is
being ETL'ed into our unified coverage datastore, but does not
yet show in the UI<br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)">* </span><span class="colour" style="color:rgb(0, 128, 0)">`grcov`</span>[4] was written mcastelluccio to
process coverage artifacts much faster; reducing processing
time from hours to seconds. <span class="colour" style="color:rgb(0, 128, 0)">`grcov`</span> also
outputs multiple formats, including for coveralls.io<br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)">* </span>There is exploration
into using coveralls.io: <a href="https://coveralls.io/jobs/24085087">https://coveralls.io/jobs/24085087</a>,
the main page is <a href="https://coveralls.io/github/marco-c/gecko-dev">https://coveralls.io/github/marco-c/gecko-dev</a>.<br></div>
<div> * JSVM coverage got working today! <a href="https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce2c56eeceb1bc252c6b25a50304f725c320d5e0">https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce2c56eeceb1bc252c6b25a50304f725c320d5e0</a> <br></div>
<div> <br></div>
<div> <span class="colour" style="color:rgb(178, 34, 34)">## The Blockers</span><br></div>
<div> <br></div>
<div> These items are further behind than I would like. If you can
help, please do.<br></div>
<div> <br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)">* </span>Getting Rust to emit
coverage artifacts is important: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1335518">https://bugzilla.mozilla.org/show_bug.cgi?id=1335518</a><br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)">* </span>Continuing the hard
work of enabling C/C++ codecoverage on all our tests: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1301170">https://bugzilla.mozilla.org/show_bug.cgi?id=1301170</a><br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)"></span><br></div>
<div> <span class="colour" style="color:rgb(178, 34, 34)">## The Biggest Problem</span><br></div>
<div> <br></div>
<div> <b>**Our test suites are
unstable**</b>. Many of you may already know that many
tests fail intermittently; these "intermittents" impact code
coverage; either by breaking the test run, and preventing
coverage collection; or by changing code paths. The challenge
is automating the metatdata collection and having a strategy
to mitigate the loss of data. [8] <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1337241">https://bugzilla.mozilla.org/show_bug.cgi?id=1337241</a><br></div>
<div> <br></div>
<div> Even so, <b>**test runs are not
deterministic**</b>, so we expect fluctuation in the
aggregate coverage statistics; How large this fluctuation is
we do not know.<br></div>
<div> <br></div>
<div> Got ideas? You want to work on a problem that no one will
thank you for? ;) Please talk to me! [3]<br></div>
<div> <br></div>
<div> <span class="colour" style="color:rgb(178, 34, 34)">## The Current Plan</span><br></div>
<div> <br></div>
<div> For the next month we will be continuing work on:<br></div>
<div> <br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)">* </span>validating the data
and process, including adding Rust and more test suites. <br></div>
<div> * <span class="colour" style="color:rgb(128, 128, 128)"></span>Adding per-line
coverage to the CoCo UI: The aggregate numbers are good for
identifying sections of code that are not tested, but we need
line-level details to inform action. <a href="https://github.com/mozilla/moz-coco/issues/7">https://github.com/mozilla/moz-coco/issues/7</a><br></div>
<div> * Explore how coveralls.io can be used to make that job
easier <br></div>
<div> <span class="colour" style="color:rgb(128, 128, 128)">* </span>Explore how to
leverage Taskcluster-generated coverage artifacts with local
coverage tools [9] <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1350446">https://bugzilla.mozilla.org/show_bug.cgi?id=1350446</a><br></div>
<div> * We are investigating if coverage tools behave differently
when e10s is enabled, then enabling it.<br></div>
<div> * JSVM ETL pipeline work will commence: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1301174">https://bugzilla.mozilla.org/show_bug.cgi?id=1301174</a><br></div>
<div> <br></div>
<div> <br></div>
<div> <span class="colour" style="color:rgb(178, 34, 34)">## Motivation</span><br></div>
<div> <br></div>
<div> <i>*Unchanged from last email*</i> -
Knowing code coverage statistics and code coverage specifics
can help evaluate risk in code: It can show what test suites
cover what lines, it can show if new code is covered by an
existing test. There are many other exciting insights we can
extract from code coverage. But first, we must collect it.<br></div>
<div> <br></div>
<div> <span class="colour" style="color:rgb(178, 34, 34)">## Reference</span><br></div>
<div> <br></div>
<div> [1] UCOSP (Undergraduate Capstone Open Source Projects) <a href="http://ucosp.ca/">http://ucosp.ca/</a><br></div>
<div> <br></div>
<div> [2] The original plan: <a href="https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#">https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#</a><br></div>
<div> <br></div>
<div> [3] Kyle Lahnakoski email: <a href="mailto:klahnakoski@mozilla.org">klahnakoski@mozilla.org</a> irc:
ekyle on #ateam@irc.mozilla.org<br></div>
<div> <br></div>
<div> [4] grcov <a href="https://github.com/marco-c/grcov">https://github.com/marco-c/grcov</a><br></div>
<div> <br></div>
<div> [5] (Co)de (Co)verage UI preview - <a href="https://ericdesj.github.io/moz-coco-w17-preview/">https://ericdesj.github.io/moz-coco-w17-preview/</a><br></div>
<div> <br></div>
<div> [5b] CoCo is on Github - <a href="https://github.com/mozilla/moz-coco">https://github.com/mozilla/moz-coco</a><br></div>
<div> <br></div>
<div> [6] CoCo relay - <a href="https://github.com/ericdesj/moz-coco-relay">https://github.com/ericdesj/moz-coco-relay</a><br></div>
<div> <br></div>
<div> [7] Rust Coverage Tracking <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1335518">https://bugzilla.mozilla.org/show_bug.cgi?id=1335518</a><br></div>
<div> <br></div>
<div> [8] Knowing when the coverage is complete: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1337241">https://bugzilla.mozilla.org/show_bug.cgi?id=1337241</a><br></div>
<div> <br></div>
<div> [9] Coverage with local tools: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1350446">https://bugzilla.mozilla.org/show_bug.cgi?id=1350446</a><br></div>
</div>
<div><br></div>
<div><br></div>
<div><br></div>
<div><br></div>
</div>
</blockquote><div><br></div>
<div><u>_______________________________________________</u><br></div>
<div>firefox-dev mailing list<br></div>
<div><a href="mailto:firefox-dev@mozilla.org">firefox-dev@mozilla.org</a><br></div>
<div><a href="https://mail.mozilla.org/listinfo/firefox-dev">https://mail.mozilla.org/listinfo/firefox-dev</a><br></div>
</blockquote><div><br></div>
</body>
</html>