Invitation for technical discussion on next-generation Thunderbird

[Joshua Cranmer 🐧]

Now is perhaps as good a time as any other to start thinking about 
technical hurdles to preparing the modules of Thunderbird.next (or 
whatever name people wish to call it). Here are some things that we are 
likely to need to start to come to a consensus on before getting any 
serious headway in coding:


    JS support and platforms

Obviously, we need a minimum baseline to support all the platforms that 
we wish to support. Something like 
<https://developer.mozilla.org/en-US/docs/Using_CXX_in_Mozilla_code> is 
a natural way to display and organize this information--you make a list 
of new and upcoming features, what the minimum version of the various 
platforms is to support them, and you slowly move the bar higher as you 
require newer versions. It also lets you see what requiring newer 
versions gets you in terms of being able to improve baseline support.

For a starting point, I think it makes sense to start with baseline of 
ES6, perhaps with async/await support. There is one major problem 
though: ES6 modules, while technically supported by all JS engines at 
this point, lack a specification for how to actually locate and load 
modules, with the current status that they're pretty much unsupported by 
everyone at this point. While I think everyone agrees that it is the 
eventual solution, it looks like it will be at least a year before 
there's usable support in any environment. And, as time has told in the 
past before, it's best to not expect an "imminent implementation" until 
there is a preview version of it actually shipping.


    Testing

Especially since JS lacks the ability to let the compiler catch common 
errors, a thorough test suite is absolutely necessary, and needs to be 
considered from day one. There's effectively three kinds of tests: 
low-level unit tests that test an individual component (e.g., IMAP 
protocol code) in isolation of a larger system, a high-level UI testing 
that drives the application from the top (à la Mozmill today), and 
system integration tests that might make sure that things like mailto: 
links work correctly or tests code that relies on changing settings that 
actually affect the computer (e.g., GSSAPI or system keyring issues). We 
don't really have any tests of the latter kind today, and it's worth 
noting that it is painful to test that kind of code when the user's 
system might be acted upon. I've suggested in the past using Docker to 
help with this, and it looks like there is some sort of support for now 
running Windows guests in a Docker container, although it's not clear if 
such support will involve monetary outlays in the future. As for UI 
testing, I have no real experience with anything here and so have no 
real comments to give. For low-level unit tests, I've come to like the 
mocha tests (although in TDD style, since I prefer suite/test over 
describe/it for some reason--note that TDD is not the default) as a good 
testing framework, and this seems to be a fairly common one in the JS 
community.

Another related but completely different aspect of testing is 
performance tests. Performance benchmarks are notoriously hard to get 
right, but it's increasingly critical these days for major applications 
(and Thunderbird certainly has a large enough userbase to be a major 
application!). Especially since JS has a not-entirely-undeserved 
reputation for being a slow language, I think it is crucial that we get 
started with performance testing very early, and we set strict 
guidelines for performance targets (I use the example of 1M message 
folders or 10-20MB email messages to guide my idea of what large things 
we might have to reasonably deal with, and these definitely give high 
constraints for code).


    Library fragmentation and sharing code

I don't know a good pithy name for this, but it's fairly clear I expect 
that everyone thinks we should be dividing the implementation into 
smaller modules/packages/etc. of some size. I think it's a good idea to 
set guidelines for where we expect these package boundaries to lie, and 
particularly on what we should do with those kinds of functions that 
might feel too low-level to ask for a dependency but crop up in many 
packages (base64 and email address manipulation come to mind). A related 
issue is that we're likely to have to polyfill some implementations into 
stuff in base APIs. For example, we probably eventually need UTF-7 
support in text conversion, and NTLM and CRAM-MD5 require crypto 
algorithms not exposed by WebCrypto (but might be exposed by other 
crypto implementations).


    API consistency for platform-inconsistent stuff

JS has a rather narrow standard library, which means a lot of key APIs 
have no real standard way to reflect them. This means we're going to 
have sit down and start figuring out common APIs to represent what we 
need to bridge these differences--which may mean adopting one API and 
polyfilling everybody else, or it might involve writing our own API and 
polyfilling everybody. A brief list of the things I can think of:

  * FFI support (we'll need it for LDAP, OS integration, GSSAPI,
    possibly things like S/MIME, import, PGP)
  * Streams: the performance targets I have require that things like
    MIME decoding need to happen in some sort of I/O stream, so this
    concept is going to run pretty deep in our code
  * Crypto
  * Sockets
  * Charset conversion
  * Database (I think we need ACID-compliant database, and I don't think
    we want to roll our own here. Note too that "load it all into
    memory" is not going to satisfy performance requirements when
    dealing with large folders).
  * Filesystem
  * DNS, HTTP, SSL, other network support
  * Multicore (whether multithreaded, multiprocess, or both)
  * Modules (might as well include it in this list since ES6 modules
    isn't an immediately available option)


    Boring infrastructure stuff that can be surprisingly controversial

Where do you host repositories? Issue tracking? Roadmaps? Build 
instructions? Support? Documentation? Continuous integration? Mailing 
lists? Updates? What license are you going to use? What build system? 
Test infrastructure? Documentation tool? Linting? Static analysis? Some 
of this stuff can be deferred or changed after coding begins in earnest. 
Some of it can't.


There's probably more topics to think of that I haven't included in this 
post, but this should definitely be a good starting point.

-- 
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/tb-planning/attachments/20170421/7e19a545/attachment-0001.html>