Comm-central build system changes

Mike Conley mconley at mozilla.com
Wed Mar 20 15:23:31 UTC 2013


First off, thanks to you, Mark, and anybody else who's been doing the 
heroic work to keep our build system going with all of these changes in m-c.

The plan in here is sound, but I expect resistance to the idea of 
putting c-c beneath m-c. This idea bubbles up every few months, and it 
meets resistance and gets shot down every time.

I think I'd like to hear from release engineering to know how they feel 
about these proposed changes.

-Mike

On 20/03/2013 12:49 AM, Joshua Cranmer 🐧 wrote:
> At present, a few of us have been working on making the comm-central
> build system work better with mozilla-central. Since we're close to
> achieving a major milestone, I felt it's best to post a status report on
> the current effort.
>
> The short version:
> * comm-central/configure.in is on the verge of going away (patches
> requesting review this week)
> * comm-central/config/* and build/* will be going away soon (next month
> or so)
> * The way you build Thunderbird will probably changing as well
> * Some requests for discussion on going forward are included
>
> Why are making these changes? Some history will explain this.
>
> In the old days, everything was in CVS and everyone used the same build
> system without caring about anything since CVS lets you checkout an
> arbitrary subset of files. When Firefox shifted development to Mercurial
> in 2008, the CVS import didn't include all the files in CVS but rather
> the subset that were deemed necessary for Mozilla 2 [1].
>
> When comm-central was created [2], there were two feasible choices for
> handling mozilla-central: build mozilla-central under comm-central or
> vice versa; the first was the option that was taken. At the time, the
> build system was still very similar to the original build system, so
> there were no major "hacks" that were necessary to get this working. Oh,
> how times have changed.
>
> The change that is probably most responsible for making the original
> comm-central build choice regrettable is the change that made
> comm-central code go inside of libxul, which can be called a mild hack.
> And, going back and reading the original bugs where these things were
> made, this change was probably originally intended to be temporary [3].
>
> Now, with the fast pace of the moz.build changes and impending
> rewrite/obliteration of the mozilla-central build system, we stand a
> good chance of suffering extensive outages as our house of cards
> crumbles. And thus we need to take major action to reduce the amount of
> code we need to port from the mozilla-central build system.
>
> I mentally divide this work into two steps. The first step is the
> emasculation of configure--bug 846540. This set of patches harmonizes
> the configuration of comm-central with mozilla-central by injecting all
> of our build variables into the mozilla-central build system.
> Incidentally, this work would have eliminated the bustage from last week
> when the XPIDLSRCS changes landed [4]. This is the major milestone of
> which I speak: the emasculated configure works on Linux platforms, and I
> am currently tracking down issues related to OS X universal builds and
> Windows configure errors. When these patches land, the only thing
> different between the two build system changes are the un-backported
> comm-central build changes [5] and path issues.
>
> The second step of changes is the step that eliminates the entire build
> system from comm-central. There are a few options to go about this.
> Presently, to make minimal adjustments to paths, I introduce lots of
> symlinks to make comm-central appear to be both a subdirectory of and an
> overlay on top of mozilla-central. Options we could use are:
>
> 1. Move comm-central to mozilla-central/comm. This is kind of the
> default assumption I'm working towards.
> 2. Keep mozilla-central in comm-central/mozilla, but add magic to the
> build system to make everything work. This might be possible, but it
> would require some gross hacks and is probably an unstable solution: I'm
> already finding subtle bugs in this arrangement just from emasculating
> configure [6].
> 3. Option #1, but allow a build under the current configuration by
> injecting enough symlinks to make it work.
> 4. Keep mozilla-central in comm-central/mozilla, but inject symlinks to
> make comm-central appear to overlay mozilla-central.
> 5. Hold the final change hostage to merging comm-central into
> mozilla-central.
>
> Most of these options require making changes to our release engineering
> systems. Using symlinks can avoid the need for changes, but comes with
> increased fragility and would completely rule out builds on FAT
> systems--I doubt they are tenable long-term options. Opinions on the way
> to go forward here are appreciated.
>
> One additional comment I hinted at in my list of options is as follows:
> once the comm-central build system is eliminated, there would remain no
> technical barrier to actually merging comm-central and mozilla-central
> into the same repository. This is not a change that can be made without
> wider project discussion, but I think the imminent prospect of
> eliminating the build system implies that this discussion should be
> started soon. Like the final build system-elimination phase, this would
> require changes to release engineering infrastructure. Benefits of the
> change include making tree-wide changes (like the recent nsITreeView
> mega-change or the constant moz.build changes) much easier to commit, as
> well as follow-on effects resulting from infrastructure coping with the
> fact that mozilla-central builds more than one product. I mention this
> now because unresolved technical aspects that could go with this change
> (e.g., whether to merge comm-central "flat" or all in a comm/
> subdirectory) could have some influence on decisions taken while
> handling the intermediate structure of the build system.
>
> Those of you who are interested in seeing the current progress on this
> front are invited to peruse all of my submissions to try-comm-central.
>
> As always, comments, questions, opinions, and thoughts are welcome.
>
> Footnotes:
> [1] History trivia: Mozilla 2 was the codename for the version of
> Mozilla that would contain large sweeping changes to eradicate technical
> debt including such things as automatic garbage collection instead of
> reference counting, using C++ exceptions instead of rv-everywhere, and
> "outparamdel" (moving return values to be just that instead of the out
> parameter). Instead, all that we got for the real Mozilla 2 (Firefox 4)
> was a rewrite of the component manager and the abolition of the notion
> of frozen interfaces.
> [2] More trivia: the original codename for comm-central was
> "CaMaSutra"-- Calendar, Mail, Suite. Actually, this might work as an
> internal name for mailnews non-Gecko internals in lieu of rkent's Skink
> name.
> [3] Interesting retrospective: it seems that mozilla-central build
> system changes are the best way to convert a lackadaisical
> make-the-comm-central-build-system-better approach into a
> get-it-committed-now-I-don't-care-how approach.
> [4] Technical details: when the first set of moz.build files landed,
> Gregory was nice enough to make changes to the mozbuild infrastructure
> to properly support building comm-central. Unfortunately, what he and
> all reviewers missed was the fact that the changes only munged the bases
> of relative paths, executing all the files with the same config.status
> file. The largest difference between the two build systems was
> MOZ_LDAP_XPCOM, which wasn't used to select files to build until the
> XPIDL flags landed. When I saw the bustage, I tested the fix locally to
> make sure it worked... on the tree that already had all these changes.
> Lesson: don't test emergency bustage fixes for a build system on top of
> a tree that partially rewrites the build system.
> [5] Yes, we actually have useful features in the comm-central build
> system that mozilla-central doesn't have. The one I know of offhand is
> $(DIR_INSTALL).
> [6] The really subtle one: autoconf works out the wrong relative path
> for ../ldap/sdks/c-sdk, which is normally innocuous, but ends up causing
> the second architecture build on an OS X universal build to pick up the
> wrong cache config file.
>



More information about the tb-planning mailing list