Summary of TB QA Meeting at Mozcamp

Mike Conley mconley at
Tue Sep 11 18:47:31 UTC 2012


I took minutes during the TB QA meeting at MozCamp.

Minutes can be found here starting on line 18: <>

Here's a summary:


  * The rapid release has really changed the way we do testing.
    Community involvement has dropped off.

  * Ludo has no chance of watching all of Bugzilla, and sometimes
    somebody comes in with a patch, and we don't see it because they
    don't set the right flags, and then it gets bitrotted and we lose a
    potential contributor.

  * We need more people to be active on Bugzilla, to take action on
    bugs, to respond when somebody files a bug to get the information we
    need to fix it (if you reply within 1 day, you have a higher degree
    of probability of getting useful information for fixing it).

  * The "bubbling up" of bugs from Bugzilla to devs to being fixed is
    what needs work. People are getting frustrated when they file bugs
    and *nothing happens*. Letting people know that when they file bugs
    properly, that things happen...that's important.

  * The notion of what NEW means on Bugzilla is different from person to
    person. Some people think NEW means, "this bug is reproducable".
    Others  think it means "this bug is reproducable, and we know where
    the fix  needs to happen."

  * IDing of bugs, of prioritizing them for being fixed - we need to
    look into new and better ways of doing this.

  * There are 4 dimensions for categorizing bugs:

      * How many people are likely to be affected

      * How serious is the effect when it occurs (workaround? consequences)

      * How much effort to fix this bug?

      * How reproducible it is

  * We need some way of visualizing bugs along these dimensions, and
    then a way of consolidating that visualization into a prioritized list.

  * But generating lists is easy. We've been generating lists for YEARS.
    The problem is that we create a list, fix a few things on it, and
    then create a new list, and repeat.

  * Fixing old bugs has taken a backseat in favour of feature
    development over the past few releases

  * Ludo is concerned about performance problems in TB - his TB on his
    beefy machine is just as performant as TB 1.x was on his old crappy
    machine. That's not a good sign.

  * Irving advocates a light-weight, "agile" list that is designed to be
    easily changed iteration after iteration, based on what is happening
    at the time. The list should make it easy to visualize the
    "life-cycle" of a bug, and allow us to measure how long it takes for
    a bug to go from being identified, to being fixed. That lets us know
    how many things to put on the list, and we can adjust accordingly.

  * We need a way for developers to examine a bug, and deprioritize it
    if it is way too complicated - ie, the rewards are too slim based on
    how much work is likely involved. (Spending time thinking about the
    hard / impossible bugs is less valuable than fixing the bugs that
    we're likely to tackle and complete)

  * We're going to tackle the Papercuts list, and see what happens. If
    we can show that we can fix the items on a list, and move the ball
    forward, then we'll worry about creating the list.

  * QA people think, "We make lists and you devs ignore them!", devs
    think, "You QA people make lists, and we can't do anything with
    them!". We need to resolve this.

  * The list should be mutable. We need to be able to change our minds.
    We don't want to thrash, but we want to be able to make adjustments.

  * We need criteria to create the initial list.

*Action items*

 1. Get a list of people watching each TB component, and make sure we've
    got somebody with eyes on each one.
 2. We're going to tackle the Papercuts list (10 bugs) to prove that we
    can deal with a list.
 3. rkent(?) is going to send out a weekly status report on the
    Papercuts list, and what needs fixing, on what got fixed.
 4. When the list is empty, or close to being empty, QA is going to
    develop a process for creating the list (which includes the facility
    for devs to indicate that a bug is too difficult).
 5. We need to come up with some critiera for choosing bugs like the
    ones on Papercuts. Then QA / Support will build the list.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the tb-planning mailing list