cc-rework, final steps

Joshua Cranmer 🐧 Pidgeot18 at gmail.com
Thu Jul 18 05:03:43 UTC 2013


[Cross-posted to several groups to make sure anyone whose input is 
desirable will see it]

cc-rework, as you may or may not recall, is the attempt of comm-central 
to divorce ourselves from having to maintain a partial copy of 
mozilla-central's build system, by placing comm-central as a 
subdirectory of mozilla-central instead of the parent directory. So 
where our tree now would be comm-central/mozilla, it would be 
mozilla-central/comm after these changes.

As of right now, I have enough patches queued up to be able to make this 
build configuration work. The patches I need for this amount to:
1. Fix makefiles to account for stricter rules in mozilla-central's 
build system (presence of IS_COMPONENT is the only one I'm aware of at 
the moment) [This can done prior to the switch].
2. Automatically rewrite paths in Makefiles and jar.mn files to account 
for the change in the top-source directory.
3. Fix paths that are not touched by those automatic rules.
4. Make some rules (like testsuite targets) work when COMM_BUILD is not 
defined [This sounds like it could be done before the switch, but it 
really can't].
5. Remove the remnants of the current build system in comm-central 
(client.mk, config/*, configure.in, top-level Makefile/moz.build, a few 
things from build/*). Not necessary, but clean up is good.

With all of these applied, I can not only build a working Thunderbird on 
Linux, I can also run through all the build steps that buildbot uses and 
get more or less the same results--such as a working installer and a 
working test package. This means that the biggest blocker to doing this 
is to make all of the necessary changes to the releng configs to get 
this to work properly. I suspect only the builders and the l10n 
packaging would need changes, largely due to shifts in paths that affect 
where things need to be checked out.

Now one change which I have not yet attempted to make is client.py. The 
projects that use comm-central can need up to 6 repositories: 
comm-central, mozilla-central, ldap/sdks, extensions/irc, 
extensions/inspector, and extensions/venkman (the latter three are used 
by SeaMonkey but not Thunderbird). The present form of getting the 
source code amounts to the following steps:
$ hg clone http://hg.mozilla.org/comm-central/
$ cd comm-central && python client.py checkout

Under the final form as envisioned by cc-rework, these steps are no 
longer necessarily sufficient. Here are several options and my thoughts 
on their tenability:
A. Keep client.py largely the same and expect users to set directories 
up correctly:
$ mkdir mozilla && cd mozilla
$ hg clone http://hg.mozilla.org/comm-central comm
$ cd comm && python client.py checkout

This requires as little changes as possible, but requires users to know 
to correctly name the comm-central directory, which requires checking 
out comm-central to a non-default repository name. It also means that 
client.py has slightly greater risk to screw you over if you don't 
follow steps properly.

B. A variant of A, where client.py does not attempt to update 
mozilla-central.
$ hg clone http://hg.mozilla.org/mozilla-central mozilla
$ hg clone http://hg.mozilla.org/comm-central mozilla/comm
$ cd mozilla/comm && python client.py checkout

Now client.py won't ever attempt to touch .., which gives it safer error 
handling (at worst, it only affects the current directory). On the 
downside, mozilla-central must be manually updated when updating a 
currently existing checkout in addition to client.py checkout.

C. Nuke client.py, and expect users to update repositories manually.

Since a regular comm-central requires effectively the tips of three 
build systems [1], one of which updates almost never. However, people 
who use all three of the other extensions need to remember to update 
more repositories buried far deeper in the tree, even if they update on 
relatively weak velocities.

D. Merge the code into fewer repositories, and combine with any of the 
above variants.

The big downside here is that merging risks causing a political 
/avalanche de merde/, particularly if it is the 
comm-central-into-mozilla-central one. The three extension repositories 
have some benefits to remaining separate repositories, in that it allows 
them to update at faster velocities than riding the trains to release; 
this is much harder to maintain if merged into either comm-central or 
mozilla-central.

It turns out that we appear to have roughly 2.5 repositories that update 
on a frequent basis [2]: mozilla-central and comm-central both see 
several checkins a day on average. The DOM Inspector addon sees the most 
development of the other four repositories, followed by Chatzilla, while 
the Venkman addon and the LDAP C SDK have seen basically no development 
in years. Since Venkman relies on an API that is slated to be removed 
(and there are no signs of developers willing to pitch in to move it to 
the new API) and is largely obsoleted by recent chrome debugging 
efforts, I would anticipate much pushback on moving it into 
mozilla-central. I don't think many people would object to merging the 
other two extensions into mozilla-central (they're too small to really 
make much of a difference), but I don't know if the maintainers of those 
add-ons are willing to see them merge into mozilla-central or not. I'm 
against merging the LDAP SDKs in for other reasons [3]. Merging 
comm-central into mozilla-central is something that needs its own, full 
discussion anyways, so I'm not going to suggest this as an immediate 
alleviation measure.

E. Merge the comm-central checkout code into mozilla-central's client.py.
$ hg clone http://hg.mozilla.org/mozilla-central mozilla
$ cd mozilla && python client.py checkout-comm

This would require some approval from mozilla-central's build peers, and 
the current use of mozilla-central's client.py is largely to update 
external repositories that are checked into mozilla-central like NSS, 
NSPR, etc. However, I do not think it would be difficult to get such 
approval for this action, and this has roughly minimal changes from the 
current workflow for obtaining or updating the build tree.

F. Like E, but use mach instead
$ hg clone http://hg.mozilla.org/mozilla-central mozilla
$ cd mozilla && ./mach checkout-comm

... Unless build peers would rather I use mach instead of client.py. 
Since releng currently uses client.py to prepare source copies for 
building, this would be the first use of mach in production releng that 
I am aware of, and I don't know how much I trust throwing that switch 
right now.


Personally, I think the only tenable long-term solution is to merge 
everything into mozilla-central. As it stands right now, we are 
effectively suffering from the submodules repository problem, and we 
currently make a wonderful case study on how annoying this problem can 
be. However, this is a potentially contentious decision, and earlier 
discussions with principals leads to the conclusion that this step, if 
taken at all, should be done as a separate step from the cc-rework step. 
Since that is out of the question for the moment, options E or F seem to 
me to be the best way to go.


Thoughts/questions/comments/concerns/flames?

[1] Actually, LDAP C-SDKS is checked out to a specific revision. 
However, that revision has effectively been tip for the existence of 
LDAP SDKs in a mercurial repository, so the distinction is minor.
[2] This list, believe it or not, is completely ordered by the date of 
the last substantive checkin--those that exclude build fixes or 
compatibility updates. In particular, Venkman's last change was January 
2012, while LDAP's last change was December 2010.
[3] I gave a lot of plausible future paths for LDAP support in 
Thunderbird in a very detailed blog post a few months ago. Since the 
only way to have a shot at decent testing is to use OpenLDAP as a 
server, if we integrate and LDAP code into comm-central, I'd rather 
integrate OpenLDAP instead of the current C SDKs.

-- 
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/tb-planning/attachments/20130718/101e729e/attachment.html>


More information about the tb-planning mailing list