comm-central future as a branch

Magnus Melin mkmelin+mozilla at iki.fi
Tue Oct 9 08:06:45 UTC 2018


I'd like feedback on the below plan for having Thunderbird in a single 
repository.
Let's have follow-ups go to dev-planning

TL;DR version: planning to have current comm-central live as a branch 
instead of a separate repository.

---

Background
----------------

When the Mozilla move from CVS to hg took place in 2008, the code 
related to Thunderbird was not included in mozilla-central and instead 
placed in the comm-central repository. Under the hood Thunderbird uses 
the mozilla-central code, so to build it currently has to check out both 
repositories. This is not ideal, for many many reasons.

I wrote [an email to 
maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html) 
last year listing some of the drawbacks of this setup - but essentially 
what it comes down to is that version control was not meant to be used 
like this and it makes workflows error prone, in addition to having to 
do various hacks in the build automation. Enabling autoland, phabricator 
and other tools used by Mozilla core could also be done much more easily 
for comm-central with a single repository.

The mail spurred some related discussion on git vs hg and importing of 
history, but basically there was agreement among the comm-central 
developers to go ahead with this change.

Thunderbird has now migrated its build automation to Taskcluster, and 
also moved to building with mozilla-central as topdir, with a comm/ 
sub-folder containing comm-central code. (We used to instead have a 
mozilla subdir checkout of mozilla-central as child the comm-central 
checkout.)

In 2014 there was [a proposal to merge comm-central code into 
mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ) 
but this was rejected since the powers that be did not want 1) the 
confusion about what code belongs to what project, 2) pull/tree size, 
and 3) it was seen as irrelevant to Firefox. The proposal below has none 
of these obstacles, and depending on the physical repository location, 
may not change anything at all for a Firefox developer.

I think it's now time to revisit the single repository idea and pull 
that through in one way or the other. So, on to the proposal. For 
reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB), so 
size-wise it is comparatively small, with a mozilla-central checkout 
weighing in at 2.3 GB. Including the relevant CVS history, the comm/.hg 
would be 317 MB in total.

How
------

To capture the essence of what Thunderbird is, we would make 
comm-central a named branch of the mozilla-central. Let's call this 
branch "comm/default".

Initially, we'd pull in a hg converted version of the current 
comm-central default branch, putting all the files into a comm 
sub-folder in the base directory of the comm/default branch. This is the 
way that the Thunderbird build system already wants the code laid out. 
With the comm sub-folder it will be clear to everyone involved which 
code is on mozilla-central code and which is not. It is possible to 
import the CVS history for the related directories too while we're at 
it, but it adds some size. Importing related CVS history only adds 
around 125 MB extra so I'm inclined to include it. With the new 
comm/default branch pulled in, we merge the curent mozilla-central 
default branch to comm/default. The resulting code is now the same as 
status quo (of the both repos pulled in), just that it's all version 
controlled in the same repository.

Once this is all set up, Thunderbird development work would move on on 
the comm/default branch, and there would be periodical merges with the 
(mozilla-central) default branch. These merges could be automated if wanted.

Quite often it's the case that Thunderbird code needs to be adjusted to 
build, due to changes in mozilla-central code. For cases of known 
incoming bustage Thunderbird now would have the possibility of waiting 
to do the merge until a fix is available. I would suggest never to back 
out bustage causing mozilla-central changesets from the comm/default 
branch, but to detect other incoming bustage by doing builds from a 
comm/band-aid branch (branced from comm/default). The details on this 
can be discussed later though, not to derail discussion about the main 
issue here. There are many alternatives, and it's also completely 
possible to do what we currently do.

Where
---------

In the earlier discussions i's become clear there is a bunch of 
confusion around what is a branch and what is a repository, so please 
note the difference. Aa repository is the physical location where the 
history of a project is stored. Every repository does not have to carry 
every branch. I.e., the mozilla-central repository could carry the 
comm/default branch, or the comm/default branch could exist only in a 
comm-central2 repository somewhere else. In all cases it's trivial for 
developers who have mozilla-central checked out, to add the comm/default 
branch to their local checkout.

So for the location there are at least three options:

A: branch in the mozilla-central repository
B: branch in the mozilla-unified repository
C: branch in a repository elsewhere

Not to disrupt operations too much I think it would be preferable to use 
option B - create the branch in the mozilla-unified repository. It does 
depend a bit on what plans Mozilla has for these repositories.

Creating the branch in mozilla-unified of course needs buy-in from the 
Mozilla hg people, so please let me hear your opinions.

What about history?
--------------------------

For the proposed approach (going through `hg convert`) commits are 
preserved, but AFAIK it's not possible to preserve actual commit hashes. 
`hg convert` will graft the original commits, and this adds the original 
hash as an extra field to the new commit (use `hg log --debug` to see 
it). I don't think this is such a big problem. The original commits 
would be linked in the pushlog, the same way there is "converted from" 
for mozilla-central-cvs, like 
https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959. 
I don't see them e.g. in 
https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611 
so perhaps there is some server side feature that needs turning on?

Can I try it?
---------------

You can check 
https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which 
carries the comm/default branch.

In your mozilla tree, pull in the branch like this:

      hg pull -u -b comm/default 
https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central

To completely remove it again, use `hg strip "branch(comm/default)`

How did you do it?
------------------------

     CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
     CC_HG_REPO=~/Code/tb/mozilla/comm/

     git clone https://github.com/ehsan/mozilla-cvs-history.git 
$CVS_GIT_REPO

     echo default comm/default | tee branchmap.txt
     echo rename . comm | tee filemap.txt cc0.txt

     hg --cwd=$CC_HG_REPO up 0
     (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /') 
 >> cc0.txt

     hg convert --filemap=cc0.txt --branchmap=branchmap.txt 
$CVS_GIT_REPO single-repo/

     CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
     CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
     echo $CC_0 $CVS_TIP | tee splicemap.txt
     # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa 
93dd94bad983ce2eff5f228719c349b78a8a7f7d

     hg convert --config convert.hg.saverev=True 
--splicemap=splicemap.txt --filemap=filemap.txt 
--branchmap=branchmap.txt $CC_HG_REPO single-repo/

For fun, see all the history down to 1999 is there, like

     hg log comm/mailnews/mime/src/mimei.cpp

Then go on and add the default branch from mozilla-central and do the merge

     cd single-repo
     hg pull -f -b default $CC_HG_REPO/..

     hg up comm/default
     hg merge default && hg commit -m "Merge default to comm/default branch"

     hg push -f -b comm/default --new-branch thunderbird-central-push

---

Thoughts, comments, feedback appreciated!

  -Magnus



More information about the tb-planning mailing list