Invitation for technical discussion on next-generation Thunderbird
ben.bucksch at beonex.com
Sat Apr 22 03:45:13 UTC 2017
Joshua Cranmer 🐧 wrote on 22.04.2017 03:52:
> Now is perhaps as good a time as any other to start thinking about
> technical hurdles to preparing the modules of Thunderbird.next (or
> whatever name people wish to call it). Here are some things that we
> are likely to need to start to come to a consensus on before getting
> any serious headway in coding:
> 0.1. JS support and platforms
> Obviously, we need a minimum baseline to support all the platforms
> that we wish to support. Something like
> is a natural way to display and organize this information--you make a
> list of new and upcoming features, what the minimum version of the
> various platforms is to support them, and you slowly move the bar
> higher as you require newer versions. It also lets you see what
> requiring newer versions gets you in terms of being able to improve
> baseline support.
* optional type checks including classes
* => functions (for async, to solve the |this| problem)
* async functions
* npm module support and platform APIs (see below)
I'd like the option to run on the following platforms, at least in
Electron uses current Chromium.
Cordova can use either the system's web browser as runtime, or
bundle an engine, and we might do the latter, to avoid strange bugs
on Android 4.x or only on some phones.
* Gecko (mostly for historical reasons)
JS in Gecko is obviously fairly advanced
What I get:
* |class| is in both Chrome and Gecko. => HAVE
* type checks. Not there yet, I think. Could use TypeScript, but that
has other drawbacks. Can emulate type checks with "assert(foo
instanceof Foo);" at the start of public functions, as poor man's
type checks. Hopefully we'll get them with later JS engines. => EMULATE
* => functions are in both Chrome and Gecko. => HAVE
* async functions: await is in Gecko 52 and Chrome 55 => HAVE
* modules. Missing in language. But we have require() from npm. Easy
to use, works well, and gives us easy access to tons of code from
npm. I don't see that a native module system would gain us a lot on
top of that. If native modules come later, and it's better, and it's
integrated with npm, we can still switch to it => EMULATE
There's a critically important feature of the platform that I really
need: Direct function calls from UI to backend modules. I want to build
a rich application API, and I don't want to explicitly marshall
everything, but use normal JS function calls. That means either shared
objects between backend and UI, or an automatic and transparent IPC
system. I understand that's technically hard, but that's an absolute
platform requirement for me.
> 0.2. Testing
> Especially since JS lacks the ability to let the compiler catch common
> errors, a thorough test suite is absolutely necessary, and needs to be
> considered from day one.
I agree we need that. But with measure. You can easily spend 40-80% of
the coding time on tests alone. I know I have, in some projects. Writing
tests should not need more than 20% of coding time, or actually 0% more
(see below). To achieve that, the test suite needs to be very
comfortable to work with.
My measure would be: If you want to test your code, instead of opening
the UI in every coding iterating, write a little test function that
calls your function, and run that. Essentially, you save yourself the
time of clicking through the UI every time, and instead use that same
time to write the test. But no more. You do not write tests just for the
sake of tests.
I can't emphasize enough how important it is to get the right measure
here. Wrong emphasis, and the project takes 5 times as long, i.e. 15
years instead of 3 years. I've seen that happen. If we do that, we won't
See also below.
> There's effectively three kinds of tests: low-level unit tests that
> test an individual component (e.g., IMAP protocol code) in isolation
> of a larger system, a high-level UI testing that drives the
> application from the top (à la Mozmill today), and system integration
> tests that might make sure that things like mailto: links work
> correctly or tests code that relies on changing settings that actually
> affect the computer (e.g., GSSAPI or system keyring issues). We don't
> really have any tests of the latter kind today, and it's worth noting
> that it is painful to test that kind of code when the user's system
> might be acted upon.
Yes. Many projects are inherently a complete system, e.g. client/server.
Ours is not - we're an email *client*. We interact with servers not
under our control, but our users totally depend on the functioning of
our client with these servers. We cannot allow even one day of gmail.com
IMAP not working. Right now, we rely on anecdotal bug reports, and it
takes time until we realize that the server changed. That needs to change.
I'd like an "integration" test suite that tests our components (not UI),
but with real-world servers. We'll set up test accounts at all ISPs that
have more than 1% market share among our users. Then we regularly run
tests (at least every 6 hours or so, and after every commit) on all of
these accounts, whether log in works, whether I get notified of new mail
(IMAP IDLE/push), I can fetch emails, send emails, copy mails etc.. If
that fails without code change, we know immediately something changed at
e.g. Gmail. If that fails after a code change, we know we broke e.g. Yahoo.
I think this high-level integration test is far more important for
Thunderbird users than classic unit tests on artificial local test
servers. We'd test what the users actually use. We'd implicitly test all
the code that's actually run when users use TB. And it's much faster to
write, because one high-level function triggers a lot of low-level code.
> Another related but completely different aspect of testing is
> performance tests. Performance benchmarks are notoriously hard to get
> right, but it's increasingly critical these days for major
> applications (and Thunderbird certainly has a large enough userbase to
> be a major application!). Especially since JS has a
> not-entirely-undeserved reputation for being a slow language, I think
> it is crucial that we get started with performance testing very early,
> and we set strict guidelines for performance targets (I use the
> example of 1M message folders or 10-20MB email messages to guide my
> idea of what large things we might have to reasonably deal with, and
> these definitely give high constraints for code).
Here I'll say: Let's first write something that can be tested. We can
always add perf tests later. There's no point starting with perf tests
early on, because adding features will necessarily slow things down in
the ms area.
I do think that perf is very important, though. In JS, there are 3 major
1. Using popular libraries. They are typically large. Many are not
written in a performant way at all (jQuery *cough*). That's #1
source for slowness in JS. Slowdown by factor 100, often.
2. Excessive DOM changes, and causing unnecessary layout "reflows".
3. "Stupid coding". This is usually obvious to an experienced developer
and caught in review or easy to fix. E.g. using strings for function
calls (jQuery *cough*), fetching too much data etc.
> API consistency for platform-inconsistent stuff
> JS has a rather narrow standard library, which means a lot of key APIs
> have no real standard way to reflect them. This means we're going to
> have sit down and start figuring out common APIs to represent what we
> need to bridge these differences--which may mean adopting one API and
> polyfilling everybody else, or it might involve writing our own API
> and polyfilling everybody. A brief list of the things I can think of:
> * FFI support (we'll need it for LDAP, OS integration, GSSAPI,
> possibly things like S/MIME, import, PGP)
TB:NG should be pure JS, with the exception of the OS integration you
list below. If some protocol implementation does not exist yet, but it's
possible, e.g. LDAP, we'll write it.
For those things that absolutely do need native calls, because of e.g.
Kerberos credentials, we might still not come to the IPC question at
all, because there's already a JS wrapper on npm, e.g.
Let's cross that bridge when we get there.
> * Filesystem
> * Sockets
> * DNS
This is all integral part of the node.js/npm platform, e.g. require("fs").
Remember that if we use existing npm libraries, they will already
presume certain npm dependencies. E.g. if we use emailjs.js IMAP
implementation, it already uses a certain socket module, likely the
native node.js one. We should look out that they are somewhat consistent
(not 13 different socket implementations by various libraries), but I
don't see a point in rewriting library code to use our artificial wrapper.
Essentially, there are already so many JS libraries on npm, and they
give so much value, that at this point, I'd make npm support a
requirement for any platform that we'll use. Including the platform APIs
they use, e.g. node.js require("fs") and sockets etc. That's basically a
requirement for all the already existing code.
So, it's hard to make such decisions in isolation. We should first get
our feet wet and make a prototype, and gain some experience with the new
situation, and then we can derive coding standards and platform APIs
Essentially, I don't want to re-do NSPR in JS. That was probably
Mozilla's biggest mistake, to write their own custom platform
abstraction. It was a necessity back then, and maybe there was nothing
better, but it led to the arcane platform it is today. Let's try to stay
in sync with the existing JS community, and use the same platform APIs.
That gives basis for collaboration, in both directions.
> * Database (I think we need ACID-compliant database, and I don't
> think we want to roll our own here. Note too that "load it all
> into memory" is not going to satisfy performance requirements when
> dealing with large folders).
Again, let's cross that bridge when we get there. For starters, I just
need preferences, mbox, and JSON. That gets me most of my way.
In some other XUL project, we used sqlite, and we eventually replaced it
module by module with simpler solutions, and ended up with no DB at all.
> 0.3. Boring infrastructure stuff that can be surprisingly controversial
> Where do you host repositories? Issue tracking? Roadmaps? Build
> instructions? Support? Documentation? Continuous integration? Mailing
> lists? Updates? What license are you going to use? What build system?
> Test infrastructure? Documentation tool? Linting? Static analysis?
> Some of this stuff can be deferred or changed after coding begins in
> earnest. Some of it can't.
We should not even going to discuss this at this point, and just
delegate this to a person to "set up our servers". Having a community
discussion on this is controversial and a waste of time for everybody,
or "rathole" as Gerv rightly called it.
Just one meta thing: We should not require contributors to make accounts
on third party servers, nor use any proprietary software. (I'll make a
single exception for github, because so much of the community is already
there, and even that could be replaced with gitlab later.)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the tb-planning