Summary of TB QA Meeting at Mozcamp
mconley at mozilla.com
Tue Sep 11 18:47:31 UTC 2012
I took minutes during the TB QA meeting at MozCamp.
Minutes can be found here starting on line 18:
Here's a summary:
* The rapid release has really changed the way we do testing.
Community involvement has dropped off.
* Ludo has no chance of watching all of Bugzilla, and sometimes
somebody comes in with a patch, and we don't see it because they
don't set the right flags, and then it gets bitrotted and we lose a
* We need more people to be active on Bugzilla, to take action on
bugs, to respond when somebody files a bug to get the information we
need to fix it (if you reply within 1 day, you have a higher degree
of probability of getting useful information for fixing it).
* The "bubbling up" of bugs from Bugzilla to devs to being fixed is
what needs work. People are getting frustrated when they file bugs
and *nothing happens*. Letting people know that when they file bugs
properly, that things happen...that's important.
* The notion of what NEW means on Bugzilla is different from person to
person. Some people think NEW means, "this bug is reproducable".
Others think it means "this bug is reproducable, and we know where
the fix needs to happen."
* IDing of bugs, of prioritizing them for being fixed - we need to
look into new and better ways of doing this.
* There are 4 dimensions for categorizing bugs:
* How many people are likely to be affected
* How serious is the effect when it occurs (workaround? consequences)
* How much effort to fix this bug?
* How reproducible it is
* We need some way of visualizing bugs along these dimensions, and
then a way of consolidating that visualization into a prioritized list.
* But generating lists is easy. We've been generating lists for YEARS.
The problem is that we create a list, fix a few things on it, and
then create a new list, and repeat.
* Fixing old bugs has taken a backseat in favour of feature
development over the past few releases
* Ludo is concerned about performance problems in TB - his TB on his
beefy machine is just as performant as TB 1.x was on his old crappy
machine. That's not a good sign.
* Irving advocates a light-weight, "agile" list that is designed to be
easily changed iteration after iteration, based on what is happening
at the time. The list should make it easy to visualize the
"life-cycle" of a bug, and allow us to measure how long it takes for
a bug to go from being identified, to being fixed. That lets us know
how many things to put on the list, and we can adjust accordingly.
* We need a way for developers to examine a bug, and deprioritize it
if it is way too complicated - ie, the rewards are too slim based on
how much work is likely involved. (Spending time thinking about the
hard / impossible bugs is less valuable than fixing the bugs that
we're likely to tackle and complete)
* We're going to tackle the Papercuts list, and see what happens. If
we can show that we can fix the items on a list, and move the ball
forward, then we'll worry about creating the list.
* QA people think, "We make lists and you devs ignore them!", devs
think, "You QA people make lists, and we can't do anything with
them!". We need to resolve this.
* The list should be mutable. We need to be able to change our minds.
We don't want to thrash, but we want to be able to make adjustments.
* We need criteria to create the initial list.
1. Get a list of people watching each TB component, and make sure we've
got somebody with eyes on each one.
2. We're going to tackle the Papercuts list (10 bugs) to prove that we
can deal with a list.
3. rkent(?) is going to send out a weekly status report on the
Papercuts list, and what needs fixing, on what got fixed.
4. When the list is empty, or close to being empty, QA is going to
develop a process for creating the list (which includes the facility
for devs to indicate that a bug is too difficult).
5. We need to come up with some critiera for choosing bugs like the
ones on Papercuts. Then QA / Support will build the list.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the tb-planning