<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 3/24/2017 12:04 PM, Ben Bucksch
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:56f75f14-d869-1ee4-9568-1abcd508a2a1@beonex.com">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<p>Summary:</p>
<ul>
<li>Our base goes away. Gecko will change dramatically in the
future, and dropping many features that Firefox does not need
anymore, but that Thunderbird relies on.</li>
<li>Our codebase is now roughly 20 years old. It heavily and
intrinsically relies on those very Gecko technologies that are
now being faded out, more or less aggressively.</li>
<li>JavaScript and HTML5 have evolved dramatically. Entire
applications in JS are now realistic, and have been done.
There are several existing JS libraries we might leverage.
JavaScript is an efficient language, which allows fast
development. A rewrite in JavaScript makes sense now.</li>
<li>We will learn from shortcomings of existing Thunderbird, and
solve them, for example a more flexible address book, and
cleanly supporting virtual folders without overhead.<br>
</li>
<li>The goal of the rewrite is to be close to the existing
Thunderbird, in UI and features, as a drop-in replacement for
end users, without baffling them. They should immediately
recognize the replacement as the Thunderbird they love. It
will install and run as normal desktop application, like
Thunderbird does today. It keeps user data local and private.<br>
</li>
<li>We can also make a new, fresh desktop UI, as alternative to
the traditional one, for new users. The technology also gives
us the option to run it as mobile app.<br>
</li>
<li>While we implement the new version of Thunderbird, the old
codebase based on Gecko will be maintained until the rewrite
is ready to replace the old one.</li>
<li>I expect this effort to take roughly 3 years: 1 year until
some dogfood (usable by some developers and enthusiasts). 2
years until a basic feature set is there. 3 years until we can
replace Thunderbird.<br>
</li>
</ul>
<p>I will describe each point in more detail below.<br>
</p>
</blockquote>
<br>
We've talked about rewriting Thunderbird for, oh, a decade or more
now. We've had agreement on the direction these rewrites should take
for at least 3 or 4 years (rkent was really one of the last
hold-outs for using JS). But we've not really made any measurable
progress on rewriting Thunderbird.<br>
<br>
The problem with rewrites is that our codebase is very
full-featured, and it's hard to replicate that functionality. And
I'm not talking about features like multipart/appledouble, which I
don't think anyone objects to throwing in a trash can. Things like
S/MIME integration, downgrading text/html to text/plain, etc. were
the ones that halted my work on JSMime. And some features require
deep integration into the interface to work properly (the DNS
PGP/SMIME key lookup functionality or NNTP's per-group password
support comes to mind, although the latter one is possibly something
to be trashed).<br>
<br>
You also have to deal with the very fundamental reality that JS,
actually, isn't all that well-suited to implementing an email
client. For any functionality that's not in the core ES spec, every
platform has a more or less fundamentally different way of
representing it. Charset conversion, crypto, TCP socket support,
even how you view generic I/O primitives are all different in each
platform, to say nothing of issues like FFI (which you'll want to
support Kerberos or LDAP). There's also the issue of email's
inherent binary/text intermingling working poorly with JS's Unicode
string/binarystring/Uint8Array confusion (it's somewhat less of an
issue in languages like C++ or Rust, where there's nop conversion
steps to treating the bytes as binary or as strings).<br>
<br>
Now, the final and biggest problem with rewrites is that there's no
backup plan if they fail. It's quite frankly inexcusable that
Thunderbird doesn't have CardDAV support yet, and there is exactly
one reason why we don't: that feature has been gated on the planned
AB rewrite for the past 10 years. We have a partially-functional
from-scratch rewrite (mconley's Ensemble project), and we even have
a CardDAV implementation for it. But when the lone developer lost
the time to work on it, all of that effort went to nought. Our
internationalized email support is something else that is being
negatively impacted by being gated on a perpetually-soon rewrite.
And with your proposal, if there's any delay in the process, if
anything takes longer than planned to implement or iron out bugs,
then delivering any improvement, even one completely orthogonal,
needs to wait in line.<br>
<br>
At the end of the day, though, that doesn't mean that there's many
other realistic options. A complete rewrite ends up being the least
bad option. The main thing I object to is your plan to
skeleton-maintain the existing TB implementation and then switch
over to brand-new TB all at once. Doing it like that runs the risk
of having the One Big Release that everyone hates (remember Vista?),
and having that after a period of basically little improvements
could prove an unrecoverable PR blow. Rather, you should think about
how to structure the rewrite so that you can import bits and pieces
of it into the existing code base as they're ready. I mean, this is
what Mozilla is doing with Firefox and Servo, so there's ample
precedent to look at. They even have the problem of "there's a giant
mess of interconnected APIs"--the CSS rendering bits can be thought
of roughly analogous to our account API. Obvious places to start are
the calendar and contacts, because those are mostly independent of
the rest of the codebase (excluding search, there's like 5 calls to
the address book, and mconley already documented the list
somewhere). In terms of other probable components that could be
moved and maintained in tandem, there's platform abstraction
(including utils like charset conversion or base64), individual
protocol implementations (roughly ns*Protocol), import, MIME,
compose, the database, search, Junk filters, the account
implementation, the front-end, and test infrastructure. Obviously,
there's some dependencies in there, but that should sum up most of
the pieces.<br>
<br>
I will point out that I have partial or complete implementations of
a lot of stuff. It would take me a while to catalog everything and
figure out what I've made public and what I haven't, but I do know
that I have:<br>
* SASL (excluding only GSSAPI at this point, because I couldn't
figure out how to test it),<br>
* a socket abstraction (the intent is to wrap a TCP socket into a
connection pool interface that supports the text/binary confusion
issue, both IMAP-style read N bytes and POP-style dot quoted, and
STARTTLS support)<br>
* an NNTP implementation on top of that socket abstraction<br>
* JSMime (my local copy is about 80% of the way to ditching
nsMsgSend and friends, i.e, I have fairly robust composition
support)<br>
* Infrastructure for running Dovecot and OpenLDAP in a Docker
container for testing purposes<br>
* a very rough basis to generate a realistic-looking dataset for
performance testing (as in, grabbing census data to find name
distributions, looking up domain name frequencies, etc.)<br>
* a proposal for a multithreaded, JS/WebWorker-centric DB API for
Thunderbird, along with an implementation of cross-thread database
notification (not trivial in WebWorkers).<br>
<br>
A final note is that, while there do exist partial implementations
of a lot stuff, it is probably better to start implementation from
scratch on a lot of stuff. Things like email.js's smtpclient.js make
a lot of assumptions in their API (e.g., "I'll only get UTF-8
strings") that aren't really safe if you want to scale to a 20M
client base. When I brought that up with the email.js list, they did
more or less agree in those deficiencies, and they were willing to
let me propose an alternative implementation to handle things that
we needed, like DSN support.<br>
<br>
In short: I think you need to give a lot more thought into breaking
a complete redesign idea into smaller modules that can be
independently reused, and you also need to think about how to import
those modules into TB in a finer-grained fashion than
all-or-nothing.<br>
<pre class="moz-signature" cols="72">--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist</pre>
</body>
</html>