<div dir="ltr">Thanks, Rob! <div><br></div><div>If you're curious, I've computed how often words like "regression", "leak", "fail", and "test" appear in backout messages, both under traditional and rapid releases:<br>
<div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra"><font face="courier new, monospace">| keyword \ cycle || traditional | rapid |</font></div><div class="gmail_extra"><font face="courier new, monospace">|-----------------||-------------|------------|</font></div>
<div class="gmail_extra"><font face="courier new, monospace">| regression || 0.09965338 | 0.02358804 |</font></div><div class="gmail_extra"><font face="courier new, monospace">| leak || 0.02599653 | 0.02923588 |</font></div>
<div class="gmail_extra"><font face="courier new, monospace">| fail || 0.1507799 | 0.2272425 |</font></div><div class="gmail_extra"><font face="courier new, monospace">| test || 0.1897747 | 0.2631229 |</font></div>
<div><br></div><div>(numbers are proportional to number of backouts; 0 = 0%, 1 = 100%)</div><div><br></div><div class="gmail_extra">You can't rely too much on word frequencies extracted from commit messages, but it appears that regressions have reduced and backouts due to test failures have increased.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">So, summarizing what you said:</div><div class="gmail_extra"><br></div><div class="gmail_extra">(a) bugs used to be fixed in-tree more often</div><div class="gmail_extra">
(b) tricky problems, like memory leaks, used to be harder to detect</div><div class="gmail_extra"><br></div><div class="gmail_extra">Regarding (b), I assume that it would contribute to a higher backout rate in the following cases:</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">(b.1) Small memory leaks, that would go unnoticed before, are now found, and the patch is backed out and fixed.</div><div class="gmail_extra">(b.2) Some memory leaks would be found much after the patch that created it was committed; as a result, a new bug report would be filled (instead of reopening the original bug report and backing out the patch)</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Is it reasonable? Of course such hypotheses would be very difficult to prove...</div><div class="gmail_extra"><br></div><div class="gmail_extra">[]s</div><div class="gmail_extra">
Rodrigo</div><br><div class="gmail_quote">On Sun, Jul 27, 2014 at 4:13 PM, Rob Campbell <span dir="ltr"><<a href="mailto:rcampbell@mozilla.com" target="_blank">rcampbell@mozilla.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word"><div>One other point occurs to me: We’ve hardened our landing processes during that timeframe. Where bugs might have had broken patches land and gotten fixed in-tree, our current process and tree sheriffs will backout obvious failures until the bugs get fixed before landing. It’s much harder to land something that is known to be “broken” in our current process than it used to be.</div>
<div><br></div><div>cheers,</div><div>Rob</div><div><div class="h5"><br><div><div>On Jul 27, 2014, at 15:10 , Rob Campbell <<a href="mailto:rcampbell@mozilla.com" target="_blank">rcampbell@mozilla.com</a>> wrote:</div>
<br><blockquote type="cite"><div style="word-wrap:break-word"><div>Hi!</div><div><br></div><div>Interesting question.</div><div><br></div><div>If I had to guess, I’d say it’s because our automated testing has improved considerably since 3.5. A number of memory leak finding tools[1] have been integrated into our test environments that are improving our early catch-rate. This is in addition to our ever-expanding set of unittests running on every checkin[2].</div>
<div><br></div><div>I think your second point ("more than 85% of backouts occurred during or before merging into central”) supports the “better testing” theory. Proving this would be difficult as you’d need to mine all of the backout-related bugs to find the reason for the backout. Searching for keywords like “regression”, “leak” or “fail” might be automatable.</div>
<div><br></div><div>~ rob</div><div><br></div><div>1- <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_memory_leaks" target="_blank">https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_memory_leaks</a></div>
<div><br></div><div>2- <a href="https://developer.mozilla.org/en-US/docs/Mochitest" target="_blank">https://developer.mozilla.org/en-US/docs/Mochitest</a></div><br><div><div>On Jul 27, 2014, at 11:12 , Rodrigo Rocha Gomes e Souza <<a href="mailto:rodrigorgs@gmail.com" target="_blank">rodrigorgs@gmail.com</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr"><div>As part of my PhD, I've been keeping track of backouts from Firefox 3.5 up to Firefox 27, and I noticed that, since the adoption of 6-week releases, the backout rate (num. of backouts / num. of bugs fixed) has increased from 6.8% to 9.4% [1]. I wonder...</div>
<div><br></div><div>1. ... have you noticed the increase in backout rate or its effects?</div><div>2. ... why would the backout rate grow under rapid release cycles?</div><div><br></div><div>Would you help me with these questions? It's always good to get feedback from people who actually work on Firefox.</div>
<div><br></div><div>Some additional insights:</div><div><br></div><div>* I noticed that, after the adoption of rapid releases, the number of bugs fixed per day increased 74%, but also that the number of developers (computed as the num. of people that committed at least 10 bug fixes) increased 75%, so *apparently* developer workload haven't increased (please correct me if I'm wrong). Therefore, workload doesn't seem to explain the growth in backout rate.</div>
<div>* I also noticed that, under rapid releases, more than 85% of backouts ocurred during or before merging into central (i.e., before the bug status changed to FIXED), versus 55% under traditional releases. Seems like an improvement, since more problems are being discovered during automated testing and less manual tests need to be repeated due to backouts.</div>
<div><br></div><div>[]s</div><div>Rodrigo</div><div><br></div><div>[1]: <a href="http://rodrigorgs.github.io/images/firefox-backouts.png" target="_blank">http://rodrigorgs.github.io/images/firefox-backouts.png</a></div></div>
_______________________________________________<br>firefox-dev mailing list<br><a href="mailto:firefox-dev@mozilla.org" target="_blank">firefox-dev@mozilla.org</a><br><a href="https://mail.mozilla.org/listinfo/firefox-dev" target="_blank">https://mail.mozilla.org/listinfo/firefox-dev</a><br>
</blockquote></div><br></div>_______________________________________________<br>firefox-dev mailing list<br><a href="mailto:firefox-dev@mozilla.org" target="_blank">firefox-dev@mozilla.org</a><br><a href="https://mail.mozilla.org/listinfo/firefox-dev" target="_blank">https://mail.mozilla.org/listinfo/firefox-dev</a><br>
</blockquote></div><br></div></div></div></blockquote></div><br></div></div></div>