<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
On 11/05/2010 05:55, Andrew Sutherland wrote:
<blockquote cite="mid:4BE8E34C.4090108@asutherland.org" type="cite"> Now
that active development is starting to happen on comm-central and
it's not just a canary for 'future problems', perhaps we should
consider actually having two sets of Thunderbird comm-central
builders:<br>
</blockquote>
I think you're right that we need to consider both alternate methods
of allowing development to continue.<br>
<blockquote cite="mid:4BE8E34C.4090108@asutherland.org" type="cite">1)
Thunderbird-canary. Always builds latest mozilla-central.
<br>
2) Thunderbird. Always builds the most recent revision of
mozilla-central that got greens on all platforms in a
Thunderbird-canary build.
<br>
</blockquote>
...<br>
<blockquote cite="mid:4BE8E34C.4090108@asutherland.org" type="cite">a)
have to run around doing heroic things to un-break the build when
they do cause comm-central-only breakage, or worse yet, request
that they back things out because of our problems.
<br>
</blockquote>
Ok, so first and foremost, we should *never* be requesting that they
back-out because of our problems. The only time I think they should
consider backing out because of problems raised by our tree is when
we've clearly picked up an issue in one of our test cases that they
haven't covered, i.e. its a true bustage that could affect Firefox
as well (for instance, I know we've picked up js faults in the
past).<br>
<br>
<blockquote cite="mid:4BE8E34C.4090108@asutherland.org" type="cite">b)
have the comm-central tree in a questionable state where we cannot
tell mozilla-central breakage from our own breakage and hence need
to close the tree.
<br>
</blockquote>
Indeed, that is the most difficult issue.<br>
<br>
So before we go into the specifics of your proposal, I'd like to
just go through some of the most frequent bustage areas that we've
been seeing:<br>
<ul>
<li>Build Config</li>
<ul>
<li>m-c has been doing a lot of rework on build config areas and
have been either breaking non-libxul builds, or changing
things such that we need to port them across to comm-central
build config.</li>
<li>These are two things that I think we can consider improving:</li>
<ul>
<li>Ted is currently working on a way to include app specific
components within the libxul library [1]. This is for
non-xulrunner builds, but would mean that we could be built
like Firefox, as mailnews could be linked into libxul *and*
use internal linkage. I probably need to cover this in more
detail elsewhere, but this would certainly help to match us
closer to FF and reduce bustage.</li>
<ul>
<li>This would also mean we could go onto using packaged
tests, which would further help with matching FF.<br>
</li>
</ul>
<li>Reworking how comm-central works. I've had on my plate for
a while (and unfortunately I've just not got round to it
yet) to start a conversation on how to improve how we do
comm-central. Basically looking at the ways of reducing the
need to port bugs from mozilla-central all the time.</li>
</ul>
</ul>
<li>Code Bustage</li>
<ul>
<li>I think these are generally less frequent, but would be
where the two sets of builders would help. These also tend to
be the ones that take longer to fix.</li>
<li>Apart from just fixing the bustages as they come along,
there isn't much we can do here.</li>
</ul>
</ul>
<br>
<blockquote cite="mid:4BE8E34C.4090108@asutherland.org" type="cite">I
would suggest that we would generally aim for Thunderbird to
generally stay within a few commits of Thunderbird-canary most of
the time. Intermittent oranges do happen, and they would likely
cause somewhat jerky motion of the revision we use, but I would
expect it to be manageable. When a comm-central-affecting commit
does land in mozilla-central, I expect we would try and resolve it
on a timescale of ~3 real days, with larger problems taking up to
a week before people we should start getting concerned. The goal
is to avoid requiring heroics or the introduction of sloppy fixes
that are created and reviewed under duress.
<br>
</blockquote>
This all sounds reasonable, especially the timescales, although I'd
still want to keep those reasonably short. I've found in the past
that when we've had bustages, grabbing the m-c person soon after the
bustage can help provide a quick solution if its not an obvious c-c
failure. This probably also falls into the category of making the
m-c person feel responsible, but I've found talking to the person
who wrote the bug valuable.<br>
<br>
My other concern with m-c bustages is having multiple bustages at
the same time - having the builders able to narrow down the bustages
to one or two changesets certainly helps.<br>
<br>
<blockquote cite="mid:4BE8E34C.4090108@asutherland.org" type="cite">I
would suggest that we do not keep the 'good' revision in revision
control, but do publish it at a public URL and that we do point
client.py at it so that random people who want to build
Thunderbird do not need to deal with breakage.
<br>
</blockquote>
Agreed, that feels like a good solution to the problem - we're not
constantly checking into revision control, and we're able to have
some sort of app that we can fine tune/poke occasionally.<br>
<br>
<blockquote cite="mid:4BE8E34C.4090108@asutherland.org" type="cite">I
would also suggest we increase the build pool size and take steps
to ensure that Thunderbird-canary cannot steal builders from
Thunderbird, as I have found it continues to do such things
currently.
<br>
</blockquote>
Again, agreed. I think balancing the builders out is one thing we
still need to work on and consider. For instance, the l10n nightly
repacks frequently steal builders. Unfortunately buildbot doesn't
have a good priority system at the moment, so we'd need our build
guy to consider the options here, but I'm sure we can find a good
solution.<br>
<br>
<br>
Mark.<br>
<br>
</body>
</html>