<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>One other point occurs to me: We’ve hardened our landing processes during that timeframe. Where bugs might have had broken patches land and gotten fixed in-tree, our current process and tree sheriffs will backout obvious failures until the bugs get fixed before landing. It’s much harder to land something that is known to be “broken” in our current process than it used to be.</div><div><br></div><div>cheers,</div><div>Rob</div><br><div><div>On Jul 27, 2014, at 15:10 , Rob Campbell <<a href="mailto:rcampbell@mozilla.com">rcampbell@mozilla.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Hi!</div><div><br></div><div>Interesting question.</div><div><br></div><div>If I had to guess, I’d say it’s because our automated testing has improved considerably since 3.5. A number of memory leak finding tools[1] have been integrated into our test environments that are improving our early catch-rate. This is in addition to our ever-expanding set of unittests running on every checkin[2].</div><div><br></div><div>I think your second point ("more than 85% of backouts occurred during or before merging into central”) supports the “better testing” theory. Proving this would be difficult as you’d need to mine all of the backout-related bugs to find the reason for the backout. Searching for keywords like “regression”, “leak” or “fail” might be automatable.</div><div><br></div><div>~ rob</div><div><br></div><div>1- <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_memory_leaks">https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_memory_leaks</a></div><div><br></div><div>2- <a href="https://developer.mozilla.org/en-US/docs/Mochitest">https://developer.mozilla.org/en-US/docs/Mochitest</a></div><br><div><div>On Jul 27, 2014, at 11:12 , Rodrigo Rocha Gomes e Souza <<a href="mailto:rodrigorgs@gmail.com">rodrigorgs@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><div>As part of my PhD, I've been keeping track of backouts from Firefox 3.5 up to Firefox 27, and I noticed that, since the adoption of 6-week releases, the backout rate (num. of backouts / num. of bugs fixed) has increased from 6.8% to 9.4% [1]. I wonder...</div>
<div><br></div><div>1. ... have you noticed the increase in backout rate or its effects?</div><div>2. ... why would the backout rate grow under rapid release cycles?</div><div><br></div><div>Would you help me with these questions? It's always good to get feedback from people who actually work on Firefox.</div>
<div><br></div><div>Some additional insights:</div><div><br></div><div>* I noticed that, after the adoption of rapid releases, the number of bugs fixed per day increased 74%, but also that the number of developers (computed as the num. of people that committed at least 10 bug fixes) increased 75%, so *apparently* developer workload haven't increased (please correct me if I'm wrong). Therefore, workload doesn't seem to explain the growth in backout rate.</div>
<div>* I also noticed that, under rapid releases, more than 85% of backouts ocurred during or before merging into central (i.e., before the bug status changed to FIXED), versus 55% under traditional releases. Seems like an improvement, since more problems are being discovered during automated testing and less manual tests need to be repeated due to backouts.</div>
<div><br></div><div>[]s</div><div>Rodrigo</div><div><br></div><div>[1]: <a href="http://rodrigorgs.github.io/images/firefox-backouts.png">http://rodrigorgs.github.io/images/firefox-backouts.png</a></div></div>
_______________________________________________<br>firefox-dev mailing list<br><a href="mailto:firefox-dev@mozilla.org">firefox-dev@mozilla.org</a><br><a href="https://mail.mozilla.org/listinfo/firefox-dev">https://mail.mozilla.org/listinfo/firefox-dev</a><br></blockquote></div><br></div>_______________________________________________<br>firefox-dev mailing list<br><a href="mailto:firefox-dev@mozilla.org">firefox-dev@mozilla.org</a><br>https://mail.mozilla.org/listinfo/firefox-dev<br></blockquote></div><br></body></html>