Proposal to start a new implementation of Thunderbird based on web technologies
Joshua Cranmer 🐧
pidgeot18 at gmail.com
Mon Apr 3 16:08:23 UTC 2017
On 3/24/2017 12:04 PM, Ben Bucksch wrote:
> * Our base goes away. Gecko will change dramatically in the future,
> and dropping many features that Firefox does not need anymore, but
> that Thunderbird relies on.
> * Our codebase is now roughly 20 years old. It heavily and
> intrinsically relies on those very Gecko technologies that are now
> being faded out, more or less aggressively.
> applications in JS are now realistic, and have been done. There
> an efficient language, which allows fast development. A rewrite in
> * We will learn from shortcomings of existing Thunderbird, and solve
> them, for example a more flexible address book, and cleanly
> supporting virtual folders without overhead.
> * The goal of the rewrite is to be close to the existing
> Thunderbird, in UI and features, as a drop-in replacement for end
> users, without baffling them. They should immediately recognize
> the replacement as the Thunderbird they love. It will install and
> run as normal desktop application, like Thunderbird does today. It
> keeps user data local and private.
> * We can also make a new, fresh desktop UI, as alternative to the
> traditional one, for new users. The technology also gives us the
> option to run it as mobile app.
> * While we implement the new version of Thunderbird, the old
> codebase based on Gecko will be maintained until the rewrite is
> ready to replace the old one.
> * I expect this effort to take roughly 3 years: 1 year until some
> dogfood (usable by some developers and enthusiasts). 2 years until
> a basic feature set is there. 3 years until we can replace
> I will describe each point in more detail below.
We've talked about rewriting Thunderbird for, oh, a decade or more now.
We've had agreement on the direction these rewrites should take for at
least 3 or 4 years (rkent was really one of the last hold-outs for using
JS). But we've not really made any measurable progress on rewriting
The problem with rewrites is that our codebase is very full-featured,
and it's hard to replicate that functionality. And I'm not talking about
features like multipart/appledouble, which I don't think anyone objects
to throwing in a trash can. Things like S/MIME integration, downgrading
text/html to text/plain, etc. were the ones that halted my work on
JSMime. And some features require deep integration into the interface to
work properly (the DNS PGP/SMIME key lookup functionality or NNTP's
per-group password support comes to mind, although the latter one is
possibly something to be trashed).
You also have to deal with the very fundamental reality that JS,
actually, isn't all that well-suited to implementing an email client.
For any functionality that's not in the core ES spec, every platform has
a more or less fundamentally different way of representing it. Charset
conversion, crypto, TCP socket support, even how you view generic I/O
primitives are all different in each platform, to say nothing of issues
like FFI (which you'll want to support Kerberos or LDAP). There's also
the issue of email's inherent binary/text intermingling working poorly
with JS's Unicode string/binarystring/Uint8Array confusion (it's
somewhat less of an issue in languages like C++ or Rust, where there's
nop conversion steps to treating the bytes as binary or as strings).
Now, the final and biggest problem with rewrites is that there's no
backup plan if they fail. It's quite frankly inexcusable that
Thunderbird doesn't have CardDAV support yet, and there is exactly one
reason why we don't: that feature has been gated on the planned AB
rewrite for the past 10 years. We have a partially-functional
from-scratch rewrite (mconley's Ensemble project), and we even have a
CardDAV implementation for it. But when the lone developer lost the time
to work on it, all of that effort went to nought. Our internationalized
email support is something else that is being negatively impacted by
being gated on a perpetually-soon rewrite. And with your proposal, if
there's any delay in the process, if anything takes longer than planned
to implement or iron out bugs, then delivering any improvement, even one
completely orthogonal, needs to wait in line.
At the end of the day, though, that doesn't mean that there's many other
realistic options. A complete rewrite ends up being the least bad
option. The main thing I object to is your plan to skeleton-maintain the
existing TB implementation and then switch over to brand-new TB all at
once. Doing it like that runs the risk of having the One Big Release
that everyone hates (remember Vista?), and having that after a period of
basically little improvements could prove an unrecoverable PR blow.
Rather, you should think about how to structure the rewrite so that you
can import bits and pieces of it into the existing code base as they're
ready. I mean, this is what Mozilla is doing with Firefox and Servo, so
there's ample precedent to look at. They even have the problem of
"there's a giant mess of interconnected APIs"--the CSS rendering bits
can be thought of roughly analogous to our account API. Obvious places
to start are the calendar and contacts, because those are mostly
independent of the rest of the codebase (excluding search, there's like
5 calls to the address book, and mconley already documented the list
somewhere). In terms of other probable components that could be moved
and maintained in tandem, there's platform abstraction (including utils
like charset conversion or base64), individual protocol implementations
(roughly ns*Protocol), import, MIME, compose, the database, search, Junk
filters, the account implementation, the front-end, and test
infrastructure. Obviously, there's some dependencies in there, but that
should sum up most of the pieces.
I will point out that I have partial or complete implementations of a
lot of stuff. It would take me a while to catalog everything and figure
out what I've made public and what I haven't, but I do know that I have:
* SASL (excluding only GSSAPI at this point, because I couldn't figure
out how to test it),
* a socket abstraction (the intent is to wrap a TCP socket into a
connection pool interface that supports the text/binary confusion issue,
both IMAP-style read N bytes and POP-style dot quoted, and STARTTLS support)
* an NNTP implementation on top of that socket abstraction
* JSMime (my local copy is about 80% of the way to ditching nsMsgSend
and friends, i.e, I have fairly robust composition support)
* Infrastructure for running Dovecot and OpenLDAP in a Docker container
for testing purposes
* a very rough basis to generate a realistic-looking dataset for
performance testing (as in, grabbing census data to find name
distributions, looking up domain name frequencies, etc.)
* a proposal for a multithreaded, JS/WebWorker-centric DB API for
Thunderbird, along with an implementation of cross-thread database
notification (not trivial in WebWorkers).
A final note is that, while there do exist partial implementations of a
lot stuff, it is probably better to start implementation from scratch on
a lot of stuff. Things like email.js's smtpclient.js make a lot of
assumptions in their API (e.g., "I'll only get UTF-8 strings") that
aren't really safe if you want to scale to a 20M client base. When I
brought that up with the email.js list, they did more or less agree in
those deficiencies, and they were willing to let me propose an
alternative implementation to handle things that we needed, like DSN
In short: I think you need to give a lot more thought into breaking a
complete redesign idea into smaller modules that can be independently
reused, and you also need to think about how to import those modules
into TB in a finer-grained fashion than all-or-nothing.
Thunderbird and DXR developer
Source code archæologist
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the tb-planning