<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 26, 2015 at 4:59 AM, Gijs Kruitbosch <span dir="ltr"><<a href="mailto:gijskruitbosch@gmail.com" target="_blank">gijskruitbosch@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">From the earlier thread about aurora uplifts on firefox-dev (I'm crossposting to m.d.platform, please follow-up on fx-dev because that's where the original thread was.):<br>
<br>
On 22/11/2015 15:41, L. David Baron wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Is a significant part of the instability on aurora related to things<br>
that land on aurora, rather than things that are unstable on nightly<br>
and then are uplifted to aurora every 6 weeks?<br>
<br>
I think if we're worried about crashes on aurora, we should be<br>
putting more resources into quickly filing and quickly fixing (or<br>
backing out) crash regressions on both nightly and aurora. My<br>
understanding is that we spend most of our topcrash-chasing<br>
resources on stages later in the release train.<br>
[...]<br>
Letting low quality hang around on nightly and aurora for extended<br>
periods of time just improves the chances that some of it will get<br>
through to release. If we want to address issues of quality on<br>
those channels, I'd rather use a project management process that<br>
looks more like sheriffing than one that looks like approvals.<br>
</blockquote>
<br>
On 23/11/2015 14:33, Robert Kaiser wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I fully agree, but we don't have the resources. My job is release quality (including stability), so I concentrate first on what is on or near release, and only as time allows take a look at Dev Edition and Nightly - which due to frequent fires on beta is very rare.<br>
In addition, we recently lost the only developer that was focused on looking at crashes (dmajor) with no replacement in sight. He looked at and found a number of issues on Nightly pretty fast, esp. as he knew what things mean when looking at code. I'm convinced we'll need someone like that again - plus additional people to look at data and identify issues.<br>
</blockquote>
<br>
What's the best way forward from this? It sounds to me like there are a number of things we could do (not necessarily exclusive):<br>
- "a project management process that looks more like sheriffing than one that looks like approvals" -- dbaron, can you expand on that? What do you have in mind?<br></blockquote><div><br></div><div>The area of questioning about channel uplift I don't see getting a lot of attention is that of how much testing was done prior to the uplift request.<br><br></div><div>How much testing did the developer do, and what were the approached taken to test?<br></div><div>How many people were engaged to help test, and what was their feedback?<br></div><div>How much time was taken to fix bugs and address the feedback we've gotten?<br><br></div><div>Too often I think we don't have good answers to these questions when the do get asked.<br></div><div><br></div><div>Nearly all the chemspill bugs continue to be bugs that were filed *before* release.<br><br></div><div>In these cases we just are just not spending enough time addressing the things that are known and being more pre-cautionary in our bugfixing.<br></div><div>Instead we tend to want to uplift and ship, then address just the blow ups.<br><br></div><div>If we are going to make progress this pattern needs to change, and we need<br></div><div>to continue to monitior, analyze, and report on what is know about the source of chemspill and regression bugs.<br></div><div> <br></div><div>The list below is a good one.<br><br></div><div>I'd add that we need to have a focused organization wide effort promote and build the number of users on all the testing channels to help increase and speed up feedback. Finding bugs is both a function of time and number of eyeballs looking at the software.<br><br></div><div>We are still stuck on about 55k for nightly, 120k on dev edition, 2.2 million on latest beta<br><a href="https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/longtermgraph/?fxnightly-adi">https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/longtermgraph/?fxnightly-adi</a><br><br></div><div>Lets brainstorm and execute on every possible thing we can do to double the number of users on each of those channels in the next six months.<br><strong><br></strong></div><div><strong><br></strong></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
- increase our focus on automated testing. I've heard suggestions that we should hire people to just write more tests, or force more landings to come with tests.<br>
<br>
- increase our use of static analysis / linting / automated scanning of things. I posted something else for the JS side of this. On the compiled code side of things, Sylvestre, do you know if any of the tools we're currently using are being run on checkins / every day to detect *newly introduced* problems specifically, and to file bugs on them (which we should be able to automate with the bugzilla API, I think)? Is there anything we can ramp up / improve there?<br>
<br>
- increase the manual testing we do of things that happen on Nightly. Ryan, can you comment on this? It seems we have the possibility to get extra QE attention for "new" projects we land on Nightly. Are there (other) people focused on quality/smoketesting things on nightly?<br>
<br>
- I notice from bugmail that we have days when new volunteers help verify fixed bugs, but this mostly happens when those bugs are on the release/beta channel at that point. How stretched are we for capacity to do that on Nightly instead? Mike, how easy would it be to adapt the tool we have for triage to give people bugs to verify? Same question for things like filing topcrashers and getting regression ranges / STR for them... Robert, are there simple instructions for how to examine and file topcrashes on MDN/wikimo? If not, would you be able to write some?<br></blockquote><div><br></div><div>There are some old articles indexed by search engines, but many probably need update<br><a href="https://www.google.com/search?q=firefox+crash+analysis">https://www.google.com/search?q=firefox+crash+analysis</a> yahoo isn't as good at finding a few of these.<br><br></div><div>We also miss really scoobiediver or someone like him<br><a href="https://www.youtube.com/watch?v=JEX-N6Wp4pM">https://www.youtube.com/watch?v=JEX-N6Wp4pM</a><br><br><strong>-chofmann</strong><br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
~ Gijs<br>
_______________________________________________<br>
firefox-dev mailing list<br>
<a href="mailto:firefox-dev@mozilla.org" target="_blank">firefox-dev@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/firefox-dev" rel="noreferrer" target="_blank">https://mail.mozilla.org/listinfo/firefox-dev</a><br>
</blockquote></div><br></div></div>