Proposal to start a new implementation of Thunderbird based on web technologies

Joshua Cranmer 🐧 pidgeot18 at gmail.com
Mon Apr 3 16:08:23 UTC 2017


On 3/24/2017 12:04 PM, Ben Bucksch wrote:
>
> Summary:
>
>   * Our base goes away. Gecko will change dramatically in the future,
>     and dropping many features that Firefox does not need anymore, but
>     that Thunderbird relies on.
>   * Our codebase is now roughly 20 years old. It heavily and
>     intrinsically relies on those very Gecko technologies that are now
>     being faded out, more or less aggressively.
>   * JavaScript and HTML5 have evolved dramatically. Entire
>     applications in JS are now realistic, and have been done. There
>     are several existing JS libraries we might leverage. JavaScript is
>     an efficient language, which allows fast development. A rewrite in
>     JavaScript makes sense now.
>   * We will learn from shortcomings of existing Thunderbird, and solve
>     them, for example a more flexible address book, and cleanly
>     supporting virtual folders without overhead.
>   * The goal of the rewrite is to be close to the existing
>     Thunderbird, in UI and features, as a drop-in replacement for end
>     users, without baffling them. They should immediately recognize
>     the replacement as the Thunderbird they love. It will install and
>     run as normal desktop application, like Thunderbird does today. It
>     keeps user data local and private.
>   * We can also make a new, fresh desktop UI, as alternative to the
>     traditional one, for new users. The technology also gives us the
>     option to run it as mobile app.
>   * While we implement the new version of Thunderbird, the old
>     codebase based on Gecko will be maintained until the rewrite is
>     ready to replace the old one.
>   * I expect this effort to take roughly 3 years: 1 year until some
>     dogfood (usable by some developers and enthusiasts). 2 years until
>     a basic feature set is there. 3 years until we can replace
>     Thunderbird.
>
> I will describe each point in more detail below.
>

We've talked about rewriting Thunderbird for, oh, a decade or more now. 
We've had agreement on the direction these rewrites should take for at 
least 3 or 4 years (rkent was really one of the last hold-outs for using 
JS). But we've not really made any measurable progress on rewriting 
Thunderbird.

The problem with rewrites is that our codebase is very full-featured, 
and it's hard to replicate that functionality. And I'm not talking about 
features like multipart/appledouble, which I don't think anyone objects 
to throwing in a trash can. Things like S/MIME integration, downgrading 
text/html to text/plain, etc. were the ones that halted my work on 
JSMime. And some features require deep integration into the interface to 
work properly (the DNS PGP/SMIME key lookup functionality or NNTP's 
per-group password support comes to mind, although the latter one is 
possibly something to be trashed).

You also have to deal with the very fundamental reality that JS, 
actually, isn't all that well-suited to implementing an email client. 
For any functionality that's not in the core ES spec, every platform has 
a more or less fundamentally different way of representing it. Charset 
conversion, crypto, TCP socket support, even how you view generic I/O 
primitives are all different in each platform, to say nothing of issues 
like FFI (which you'll want to support Kerberos or LDAP). There's also 
the issue of email's inherent binary/text intermingling working poorly 
with JS's Unicode string/binarystring/Uint8Array confusion (it's 
somewhat less of an issue in languages like C++ or Rust, where there's 
nop conversion steps to treating the bytes as binary or as strings).

Now, the final and biggest problem with rewrites is that there's no 
backup plan if they fail. It's quite frankly inexcusable that 
Thunderbird doesn't have CardDAV support yet, and there is exactly one 
reason why we don't: that feature has been gated on the planned AB 
rewrite for the past 10 years. We have a partially-functional 
from-scratch rewrite (mconley's Ensemble project), and we even have a 
CardDAV implementation for it. But when the lone developer lost the time 
to work on it, all of that effort went to nought. Our internationalized 
email support is something else that is being negatively impacted by 
being gated on a perpetually-soon rewrite. And with your proposal, if 
there's any delay in the process, if anything takes longer than planned 
to implement or iron out bugs, then delivering any improvement, even one 
completely orthogonal, needs to wait in line.

At the end of the day, though, that doesn't mean that there's many other 
realistic options. A complete rewrite ends up being the least bad 
option. The main thing I object to is your plan to skeleton-maintain the 
existing TB implementation and then switch over to brand-new TB all at 
once. Doing it like that runs the risk of having the One Big Release 
that everyone hates (remember Vista?), and having that after a period of 
basically little improvements could prove an unrecoverable PR blow. 
Rather, you should think about how to structure the rewrite so that you 
can import bits and pieces of it into the existing code base as they're 
ready. I mean, this is what Mozilla is doing with Firefox and Servo, so 
there's ample precedent to look at. They even have the problem of 
"there's a giant mess of interconnected APIs"--the CSS rendering bits 
can be thought of roughly analogous to our account API. Obvious places 
to start are the calendar and contacts, because those are mostly 
independent of the rest of the codebase (excluding search, there's like 
5 calls to the address book, and mconley already documented the list 
somewhere). In terms of other probable components that could be moved 
and maintained in tandem, there's platform abstraction (including utils 
like charset conversion or base64), individual protocol implementations 
(roughly ns*Protocol), import, MIME, compose, the database, search, Junk 
filters, the account implementation, the front-end, and test 
infrastructure. Obviously, there's some dependencies in there, but that 
should sum up most of the pieces.

I will point out that I have partial or complete implementations of a 
lot of stuff. It would take me a while to catalog everything and figure 
out what I've made public and what I haven't, but I do know that I have:
* SASL (excluding only GSSAPI at this point, because I couldn't figure 
out how to test it),
* a socket abstraction (the intent is to wrap a TCP socket into a 
connection pool interface that supports the text/binary confusion issue, 
both IMAP-style read N bytes and POP-style dot quoted, and STARTTLS support)
* an NNTP implementation on top of that socket abstraction
* JSMime (my local copy is about 80% of the way to ditching nsMsgSend 
and friends, i.e, I have fairly robust composition support)
* Infrastructure for running Dovecot and OpenLDAP in a Docker container 
for testing purposes
* a very rough basis to generate a realistic-looking dataset for 
performance testing (as in, grabbing census data to find name 
distributions, looking up domain name frequencies, etc.)
* a proposal for a multithreaded, JS/WebWorker-centric DB API for 
Thunderbird, along with an implementation of cross-thread database 
notification (not trivial in WebWorkers).

A final note is that, while there do exist partial implementations of a 
lot stuff, it is probably better to start implementation from scratch on 
a lot of stuff. Things like email.js's smtpclient.js make a lot of 
assumptions in their API (e.g., "I'll only get UTF-8 strings") that 
aren't really safe if you want to scale to a 20M client base. When I 
brought that up with the email.js list, they did more or less agree in 
those deficiencies, and they were willing to let me propose an 
alternative implementation to handle things that we needed, like DSN 
support.

In short: I think you need to give a lot more thought into breaking a 
complete redesign idea into smaller modules that can be independently 
reused, and you also need to think about how to import those modules 
into TB in a finer-grained fashion than all-or-nothing.

-- 
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/tb-planning/attachments/20170403/82493b1f/attachment-0001.html>


More information about the tb-planning mailing list