Comm-central build system changes

Joshua Cranmer 🐧 Pidgeot18 at gmail.com
Wed Mar 20 04:49:37 UTC 2013


At present, a few of us have been working on making the comm-central 
build system work better with mozilla-central. Since we're close to 
achieving a major milestone, I felt it's best to post a status report on 
the current effort.

The short version:
* comm-central/configure.in is on the verge of going away (patches 
requesting review this week)
* comm-central/config/* and build/* will be going away soon (next month 
or so)
* The way you build Thunderbird will probably changing as well
* Some requests for discussion on going forward are included

Why are making these changes? Some history will explain this.

In the old days, everything was in CVS and everyone used the same build 
system without caring about anything since CVS lets you checkout an 
arbitrary subset of files. When Firefox shifted development to Mercurial 
in 2008, the CVS import didn't include all the files in CVS but rather 
the subset that were deemed necessary for Mozilla 2 [1].

When comm-central was created [2], there were two feasible choices for 
handling mozilla-central: build mozilla-central under comm-central or 
vice versa; the first was the option that was taken. At the time, the 
build system was still very similar to the original build system, so 
there were no major "hacks" that were necessary to get this working. Oh, 
how times have changed.

The change that is probably most responsible for making the original 
comm-central build choice regrettable is the change that made 
comm-central code go inside of libxul, which can be called a mild hack. 
And, going back and reading the original bugs where these things were 
made, this change was probably originally intended to be temporary [3].

Now, with the fast pace of the moz.build changes and impending 
rewrite/obliteration of the mozilla-central build system, we stand a 
good chance of suffering extensive outages as our house of cards 
crumbles. And thus we need to take major action to reduce the amount of 
code we need to port from the mozilla-central build system.

I mentally divide this work into two steps. The first step is the 
emasculation of configure--bug 846540. This set of patches harmonizes 
the configuration of comm-central with mozilla-central by injecting all 
of our build variables into the mozilla-central build system. 
Incidentally, this work would have eliminated the bustage from last week 
when the XPIDLSRCS changes landed [4]. This is the major milestone of 
which I speak: the emasculated configure works on Linux platforms, and I 
am currently tracking down issues related to OS X universal builds and 
Windows configure errors. When these patches land, the only thing 
different between the two build system changes are the un-backported 
comm-central build changes [5] and path issues.

The second step of changes is the step that eliminates the entire build 
system from comm-central. There are a few options to go about this. 
Presently, to make minimal adjustments to paths, I introduce lots of 
symlinks to make comm-central appear to be both a subdirectory of and an 
overlay on top of mozilla-central. Options we could use are:

1. Move comm-central to mozilla-central/comm. This is kind of the 
default assumption I'm working towards.
2. Keep mozilla-central in comm-central/mozilla, but add magic to the 
build system to make everything work. This might be possible, but it 
would require some gross hacks and is probably an unstable solution: I'm 
already finding subtle bugs in this arrangement just from emasculating 
configure [6].
3. Option #1, but allow a build under the current configuration by 
injecting enough symlinks to make it work.
4. Keep mozilla-central in comm-central/mozilla, but inject symlinks to 
make comm-central appear to overlay mozilla-central.
5. Hold the final change hostage to merging comm-central into 
mozilla-central.

Most of these options require making changes to our release engineering 
systems. Using symlinks can avoid the need for changes, but comes with 
increased fragility and would completely rule out builds on FAT 
systems--I doubt they are tenable long-term options. Opinions on the way 
to go forward here are appreciated.

One additional comment I hinted at in my list of options is as follows: 
once the comm-central build system is eliminated, there would remain no 
technical barrier to actually merging comm-central and mozilla-central 
into the same repository. This is not a change that can be made without 
wider project discussion, but I think the imminent prospect of 
eliminating the build system implies that this discussion should be 
started soon. Like the final build system-elimination phase, this would 
require changes to release engineering infrastructure. Benefits of the 
change include making tree-wide changes (like the recent nsITreeView 
mega-change or the constant moz.build changes) much easier to commit, as 
well as follow-on effects resulting from infrastructure coping with the 
fact that mozilla-central builds more than one product. I mention this 
now because unresolved technical aspects that could go with this change 
(e.g., whether to merge comm-central "flat" or all in a comm/ 
subdirectory) could have some influence on decisions taken while 
handling the intermediate structure of the build system.

Those of you who are interested in seeing the current progress on this 
front are invited to peruse all of my submissions to try-comm-central.

As always, comments, questions, opinions, and thoughts are welcome.

Footnotes:
[1] History trivia: Mozilla 2 was the codename for the version of 
Mozilla that would contain large sweeping changes to eradicate technical 
debt including such things as automatic garbage collection instead of 
reference counting, using C++ exceptions instead of rv-everywhere, and 
"outparamdel" (moving return values to be just that instead of the out 
parameter). Instead, all that we got for the real Mozilla 2 (Firefox 4) 
was a rewrite of the component manager and the abolition of the notion 
of frozen interfaces.
[2] More trivia: the original codename for comm-central was 
"CaMaSutra"-- Calendar, Mail, Suite. Actually, this might work as an 
internal name for mailnews non-Gecko internals in lieu of rkent's Skink 
name.
[3] Interesting retrospective: it seems that mozilla-central build 
system changes are the best way to convert a lackadaisical 
make-the-comm-central-build-system-better approach into a 
get-it-committed-now-I-don't-care-how approach.
[4] Technical details: when the first set of moz.build files landed, 
Gregory was nice enough to make changes to the mozbuild infrastructure 
to properly support building comm-central. Unfortunately, what he and 
all reviewers missed was the fact that the changes only munged the bases 
of relative paths, executing all the files with the same config.status 
file. The largest difference between the two build systems was 
MOZ_LDAP_XPCOM, which wasn't used to select files to build until the 
XPIDL flags landed. When I saw the bustage, I tested the fix locally to 
make sure it worked... on the tree that already had all these changes. 
Lesson: don't test emergency bustage fixes for a build system on top of 
a tree that partially rewrites the build system.
[5] Yes, we actually have useful features in the comm-central build 
system that mozilla-central doesn't have. The one I know of offhand is 
$(DIR_INSTALL).
[6] The really subtle one: autoconf works out the wrong relative path 
for ../ldap/sdks/c-sdk, which is normally innocuous, but ends up causing 
the second architecture build on an OS X universal build to pick up the 
wrong cache config file.

-- 
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist




More information about the tb-planning mailing list