<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Kyle, do you know if Rust code coverage is blocked on any remaining
Rust toolchain issues?<br>
<br>
chris<br>
<br>
<br>
<div class="moz-cite-prefix">On 2017-08-10 11:31 AM, Kyle Lahnakoski
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:e41636c6-e241-8a0d-69dd-4a2eb4461c10@mozilla.com">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<p><br>
</p>
<p>Did you have that sense you were missing something? Well, you
were right: You were missing your ...<br>
</p>
<p><font size="+1"># <b>Monthly CodeCoverage! update! \o/ </b></font><br>
</p>
<p><i>If you </i><i>want to hear more about an</i><i>y of the</i><i>s</i><i>e
items, please contact me and I will get you more detailed
information</i><b><br>
</b></p>
<p><b>## </b><b>Summary of July</b><br>
</p>
<ul>
<li><b>More hard work on the ETL pipeline</b>: Condensing the
data to make it manageable, and writing more efficient code
for faster processing. There is still more work to be done,
but it's working. Marco summarized this work with an
excellent blog post: <span
class="author-a-fez78zz87ziqz68zz78zejgez84zx36 url"><a
href="https://marco-c.github.io/2017/07/28/code-coverage-architecture.html"
moz-do-not-send="true">[1]
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html</a></span><br>
</li>
<li><b>Analyzing coverage variability</b> - If you recall from
previous months, I mentioned that different coverage runs show
different coverage numbers: We call this "coverage
variability", and it even exists when comparing two runs from
the same revision. gmierz has been looking into this
variability to better understand its size, and character. [2],
[3]<br>
</li>
<li><b>Finished detailed the plan for the MVP </b>[4] - This
MVP is for visualizing coverage of net-new lines: Relman is
interested in the coverage on the net-new lines of every
changeset This is hard given we can only run coverage on
every central push. We now have a plan for how to get this
done, as best as possible, with the information given. </li>
</ul>
<p><b>## Plans for August</b></p>
<ul>
<li><b>Build the MVP</b> - Visualize coverage of net new lines
by changeset: Now that we have a plan, armenzg has already
started the frontend UI work [5], [6], and will be working
with marco working on the backend [7]<br>
</li>
<li><b>Continue work on optimizing the ETL pipelines</b> -
necessary work <b><br>
</b></li>
</ul>
<p><b><br>
</b><b>## Meetings</b></p>
<p> </p>
We have weekly CodeCoverage meetings, and you are welcome to
attend:<br>
<ul>
<li> <span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z">When:
Held every Friday @ 11:30 EDT (08:30 PDT)</span></li>
<li><span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z">Where:
Kyle's video room </span><span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z
url"><a moz-do-not-send="true"
href="https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC">https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC</a></span>
</li>
<li><span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z">Etherpad:
</span><span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z
url"><a moz-do-not-send="true"
href="https://public.etherpad-mozilla.org/p/code_coverage_Q1_17">https://public.etherpad-mozilla.org/p/code_coverage_Q1_17</a></span>
</li>
</ul>
<p><br>
<b>## Reference</b></p>
<p>[1] Blog post on coverage collection and aggregation: <span
class="author-a-fez78zz87ziqz68zz78zejgez84zx36 url"><a
href="https://marco-c.github.io/2017/07/28/code-coverage-architecture.html"
moz-do-not-send="true">https://marco-c.github.io/2017/07/28/code-coverage-architecture.html</a></span></p>
<p><span class="author-a-fez78zz87ziqz68zz78zejgez84zx36 url"></span>[2]
Variability analysis code - <a class="moz-txt-link-freetext"
href="https://github.com/gmierz/coco-variability-tools"
moz-do-not-send="true">https://github.com/gmierz/coco-variability-tools</a></p>
<p>[3] Example variability output - <a
class="moz-txt-link-freetext"
href="https://gmierz.github.io/variability/variability_index.html"
moz-do-not-send="true">https://gmierz.github.io/variability/variability_index.html</a></p>
<p>[4] Details for UI - <a class="moz-txt-link-freetext"
href="https://public.etherpad-mozilla.org/p/code_coverage_Q3_17"
moz-do-not-send="true">https://public.etherpad-mozilla.org/p/code_coverage_Q3_17</a></p>
<p>[5] Code for UI - <a class="moz-txt-link-freetext"
href="https://github.com/armenzg/code_cov_experiments"
moz-do-not-send="true">https://github.com/armenzg/code_cov_experiments</a></p>
<p> [6] Example UI (very rough, but stay tuned!) - <a
class="moz-txt-link-freetext"
href="https://armenzg.github.io/code_cov_experiments/"
moz-do-not-send="true">https://armenzg.github.io/code_cov_experiments/</a></p>
<p>[7] Code for backend -
<a class="moz-txt-link-freetext"
href="https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage"
moz-do-not-send="true">https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage</a></p>
<p><br>
</p>
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 2017-07-06 21:37, Kyle Lahnakoski
wrote:<br>
</div>
<blockquote
cite="mid:c4485ef3-884c-d28c-0100-f805ce5c91e4@mozilla.com"
type="cite">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<p> </p>
<div class="moz-text-html" lang="x-unicode"> <br>
<br>
## Summary of Past Quarter<br>
<br>
Coverage is enabled for nearly all tests, and scheduled every
push [1]!! <br>
<ul>
<li>All c/c++ test suites have coverage enabled<br>
</li>
<li>talos coverage is enabled<br>
</li>
<li>jsvm[7] coverage is enabled, and running </li>
<li>codecov.io [2] shows the results, broken down by
directory<br>
</li>
</ul>
## Plans for Q3<br>
<br>
The overall plan for Q3 is laid out in the planning document
[12]. Put simply, a **coverage differential viewer**,
operating at low resolution (per central push), has enough
promise to justify Q3 effort on CodeCoverage.<br>
<br>
## The Complications<br>
<ul>
<li>Rust code coverage is still delayed [6] - maybe by mid
quarter<br>
</li>
<li>The data volume is very large; coveralls.io and
codecov.io are having some difficulty dealing with the
volume.</li>
<li>There is instability in the coverage numbers due to
variability in our test runs; think GC and telemetry
logic. Multiple coverage runs will be required to get a
total coverage number</li>
<li>Intermittents are impacting coverage reliability - we
will require a coverage health monitor to know if coverage
is "complete"<br>
</li>
</ul>
## Summary of this past June<br>
<ul>
<li><span
class="author-a-8hz72z5ez70zz79z2fz65zz76z58z84z5l">upgrading
tests to use Ubuntu 16.04<br>
</span></li>
<li><span
class="author-a-8hz72z5ez70zz79z2fz65zz76z58z84z5l"></span>fixing
blockers that stood in the way of rust coverage<span
class="author-a-8hz72z5ez70zz79z2fz65zz76z58z84z5l"></span>
[6]<br>
</li>
<li>enabling coverage to even more suites, like talos [10] <span
class="author-a-5wz83zxfz84zz77z1z67zfck1cz75zz75z url"></span></li>
<li><span
class="author-a-8hz72z5ez70zz79z2fz65zz76z58z84z5l">adding
JavaScript to the codecov/coveralls report</span></li>
<li><span
class="author-a-8hz72z5ez70zz79z2fz65zz76z58z84z5l">getting
a handle on the volume of data code coverage is
producing<br>
</span></li>
</ul>
<p>## Plans for July</p>
<ul>
<li>continued work on ETL pipeline</li>
<li>enable coverage for spidermonkey [11]</li>
<li>see the first hints of Rust coverage</li>
<li>build a coverage health monitor to deal with "the
complications" (above)<br>
</li>
</ul>
## Meetings<br>
<p>We have weekly CodeCoverage meetings, and you are welcome
to attend:<br>
</p>
<ul>
<li> <span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z">When:
Held every Friday @ 11:30 EDT (08:30 PDT)</span> </li>
<li><span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z">Where:
Kyle's video room </span><span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z
url"><a moz-do-not-send="true"
href="https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC">https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC</a></span>
</li>
<li><span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z">Etherpad:
</span><span
class="author-a-fbz76zz67z1z74zz80zdz71zz75zkz90zqz78z6z82z
url"><a moz-do-not-send="true"
href="https://public.etherpad-mozilla.org/p/code_coverage_Q1_17">https://public.etherpad-mozilla.org/p/code_coverage_Q1_17</a></span>
</li>
</ul>
<p><br>
</p>
<p>## Reference<br>
</p>
<p>[1] See coverage on TH <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov">https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov</a></p>
<p>[1b] Example on TH: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=433c379d6e448dc64df25a6b4d8739c99e84d102&filter-searchStr=ccov">https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=433c379d6e448dc64df25a6b4d8739c99e84d102&filter-searchStr=ccov</a>
<br>
</p>
<p>[2] codecov.io <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://codecov.io/gh/marco-c/gecko-dev">https://codecov.io/gh/marco-c/gecko-dev</a>
</p>
<p>[3] Local Coverage <a moz-do-not-send="true"
rel="nofollow"
href="https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_%28or_any_other_treeherder_build">https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build</a>)</p>
[4] iOS Coverage <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master">https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master</a><br>
<br>
[5] User Cases <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4">https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4</a>-
<br>
<p>[6] Rust coverage <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1335518">https://bugzilla.mozilla.org/show_bug.cgi?id=1335518</a>
and <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://github.com/rust-lang/rust/pull/42433">https://github.com/rust-lang/rust/pull/42433</a>
</p>
<p>[7] JSVM coverage: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1301174">https://bugzilla.mozilla.org/show_bug.cgi?id=1301174</a></p>
<p>[8] e10s coverage example: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51">https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51</a></p>
[9] Everything: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393">https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393</a><br>
<p>[10] talos coverage: <span
class="author-a-5wz83zxfz84zz77z1z67zfck1cz75zz75z"></span><span
class="author-a-5wz83zxfz84zz77z1z67zfck1cz75zz75z url"><a
moz-do-not-send="true"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1372324">https://bugzilla.mozilla.org/show_bug.cgi?id=1372324</a></span></p>
<p><span class="author-a-5wz83zxfz84zz77z1z67zfck1cz75zz75z
url"><a moz-do-not-send="true"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1372324">[11]
spidermonkey coverage:
https://bugzilla.mozilla.org/show_bug.cgi?id=1362013</a></span></p>
<p><span class="author-a-5wz83zxfz84zz77z1z67zfck1cz75zz75z
url"><a moz-do-not-send="true"
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1372324">[12]
Planning document:
https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit?usp=sharing</a></span><br>
</p>
<p><a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://marco-c.github.io/grcov-test/coverage_by_dir.html"><br>
</a></p>
<p><br>
</p>
<br>
<p><br>
</p>
</div>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
firefox-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:firefox-dev@mozilla.org">firefox-dev@mozilla.org</a>
<a class="moz-txt-link-freetext" href="https://mail.mozilla.org/listinfo/firefox-dev">https://mail.mozilla.org/listinfo/firefox-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>