<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Joshua Cranmer 🐧 wrote on 22.04.2017
03:52:<br>
</div>
<blockquote
cite="mid:5a31e3ae-e621-94df-5253-30d986ac674c@gmail.com"
type="cite">
<meta http-equiv="Context-Type" content="text/html; charset=utf-8">
<p>Now is perhaps as good a time as any other to start thinking
about technical hurdles to preparing the modules of
Thunderbird.next (or whatever name people wish to call it). Here
are some things that we are likely to need to start to come to a
consensus on before getting any serious headway in coding:</p>
<h2>JS support and platforms<br>
</h2>
<p>Obviously, we need a minimum baseline to support all the
platforms that we wish to support. Something like
<a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
href="https://developer.mozilla.org/en-US/docs/Using_CXX_in_Mozilla_code"><https://developer.mozilla.org/en-US/docs/Using_CXX_in_Mozilla_code></a>
is a natural way to display and organize this information--you
make a list of new and upcoming features, what the minimum
version of the various platforms is to support them, and you
slowly move the bar higher as you require newer versions. It
also lets you see what requiring newer versions gets you in
terms of being able to improve baseline support.</p>
</blockquote>
<br>
Wishlist:<br>
What I want in the JavaScript language:<br>
<ul>
<li>|class|</li>
<li>optional type checks including classes</li>
<li>=> functions (for async, to solve the |this| problem)</li>
<li>async functions</li>
<li>modules</li>
<li>npm module support and platform APIs (see below)<br>
</li>
</ul>
<br>
Platforms:<br>
I'd like the option to run on the following platforms, at least in
principle:<br>
<ul>
<li>Electron<br>
Electron uses current Chromium.<br>
</li>
<li>Cordova<br>
Cordova can use either the system's web browser as runtime, or
bundle an engine, and we might do the latter, to avoid strange
bugs on Android 4.x or only on some phones.<br>
</li>
<li>Gecko (mostly for historical reasons)<br>
JS in Gecko is obviously fairly advanced<br>
</li>
</ul>
<br>
What I get:<br>
<ul>
<li>|class| is in both Chrome and Gecko. => HAVE</li>
<li>type checks. Not there yet, I think. Could use TypeScript, but
that has other drawbacks. Can emulate type checks with
"assert(foo instanceof Foo);" at the start of public functions,
as poor man's type checks. Hopefully we'll get them with later
JS engines. => EMULATE</li>
<li>=> functions are in both Chrome and Gecko. => HAVE</li>
<li>async functions: await is in Gecko 52 and Chrome 55 => HAVE</li>
<li>modules. Missing in language. But we have require() from npm.
Easy to use, works well, and gives us easy access to tons of
code from npm. I don't see that a native module system would
gain us a lot on top of that. If native modules come later, and
it's better, and it's integrated with npm, we can still switch
to it => EMULATE</li>
</ul>
<br>
References:<br>
<a class="moz-txt-link-freetext" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/class">https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/class</a><br>
<a class="moz-txt-link-freetext" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await">https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await</a><br>
<a class="moz-txt-link-freetext" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import">https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import</a><br>
<a class="moz-txt-link-freetext" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/export">https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/export</a><br>
<br>
<br>
There's a critically important feature of the platform that I really
need: Direct function calls from UI to backend modules. I want to
build a rich application API, and I don't want to explicitly
marshall everything, but use normal JS function calls. That means
either shared objects between backend and UI, or an automatic and
transparent IPC system. I understand that's technically hard, but
that's an absolute platform requirement for me.<br>
<br>
<blockquote
cite="mid:5a31e3ae-e621-94df-5253-30d986ac674c@gmail.com"
type="cite">
<h2>Testing<br>
</h2>
<p>Especially since JS lacks the ability to let the compiler catch
common errors, a thorough test suite is absolutely necessary,
and needs to be considered from day one.</p>
</blockquote>
<br>
I agree we need that. But with measure. You can easily spend 40-80%
of the coding time on tests alone. I know I have, in some projects.
Writing tests should not need more than 20% of coding time, or
actually 0% more (see below). To achieve that, the test suite needs
to be very comfortable to work with.<br>
<br>
My measure would be: If you want to test your code, instead of
opening the UI in every coding iterating, write a little test
function that calls your function, and run that. Essentially, you
save yourself the time of clicking through the UI every time, and
instead use that same time to write the test. But no more. You do
not write tests just for the sake of tests.<br>
<br>
I can't emphasize enough how important it is to get the right
measure here. Wrong emphasis, and the project takes 5 times as long,
i.e. 15 years instead of 3 years. I've seen that happen. If we do
that, we won't finish.<br>
<br>
See also below.<br>
<br>
<blockquote
cite="mid:5a31e3ae-e621-94df-5253-30d986ac674c@gmail.com"
type="cite">
<p>There's effectively three kinds of tests: low-level unit tests
that test an individual component (e.g., IMAP protocol code) in
isolation of a larger system, a high-level UI testing that
drives the application from the top (à la Mozmill today), and
system integration tests that might make sure that things like
mailto: links work correctly or tests code that relies on
changing settings that actually affect the computer (e.g.,
GSSAPI or system keyring issues). We don't really have any tests
of the latter kind today, and it's worth noting that it is
painful to test that kind of code when the user's system might
be acted upon.</p>
</blockquote>
<br>
Yes. Many projects are inherently a complete system, e.g.
client/server. Ours is not - we're an email *client*. We interact
with servers not under our control, but our users totally depend on
the functioning of our client with these servers. We cannot allow
even one day of gmail.com IMAP not working. Right now, we rely on
anecdotal bug reports, and it takes time until we realize that the
server changed. That needs to change.<br>
<br>
I'd like an "integration" test suite that tests our components (not
UI), but with real-world servers. We'll set up test accounts at all
ISPs that have more than 1% market share among our users. Then we
regularly run tests (at least every 6 hours or so, and after every
commit) on all of these accounts, whether log in works, whether I
get notified of new mail (IMAP IDLE/push), I can fetch emails, send
emails, copy mails etc.. If that fails without code change, we know
immediately something changed at e.g. Gmail. If that fails after a
code change, we know we broke e.g. Yahoo.<br>
<br>
I think this high-level integration test is far more important for
Thunderbird users than classic unit tests on artificial local test
servers. We'd test what the users actually use. We'd implicitly test
all the code that's actually run when users use TB. And it's much
faster to write, because one high-level function triggers a lot of
low-level code.<br>
<br>
<blockquote
cite="mid:5a31e3ae-e621-94df-5253-30d986ac674c@gmail.com"
type="cite">
<p>Another related but completely different aspect of testing is
performance tests. Performance benchmarks are notoriously hard
to get right, but it's increasingly critical these days for
major applications (and Thunderbird certainly has a large enough
userbase to be a major application!). Especially since JS has a
not-entirely-undeserved reputation for being a slow language, I
think it is crucial that we get started with performance testing
very early, and we set strict guidelines for performance targets
(I use the example of 1M message folders or 10-20MB email
messages to guide my idea of what large things we might have to
reasonably deal with, and these definitely give high constraints
for code).<br>
</p>
</blockquote>
<br>
Here I'll say: Let's first write something that can be tested. We
can always add perf tests later. There's no point starting with perf
tests early on, because adding features will necessarily slow things
down in the ms area.<br>
<br>
I do think that perf is very important, though. In JS, there are 3
major perf killers:<br>
<ol>
<li>Using popular libraries. They are typically large. Many are
not written in a performant way at all (jQuery *cough*). That's
#1 source for slowness in JS. Slowdown by factor 100, often.</li>
<li>Excessive DOM changes, and causing unnecessary layout
"reflows".</li>
<li>"Stupid coding". This is usually obvious to an experienced
developer and caught in review or easy to fix. E.g. using
strings for function calls (jQuery *cough*), fetching too much
data etc.<br>
</li>
</ol>
<br>
<blockquote
cite="mid:5a31e3ae-e621-94df-5253-30d986ac674c@gmail.com"
type="cite">
<p> </p>
API consistency for platform-inconsistent stuff
<p>JS has a rather narrow standard library, which means a lot of
key APIs have no real standard way to reflect them. This means
we're going to have sit down and start figuring out common APIs
to represent what we need to bridge these differences--which may
mean adopting one API and polyfilling everybody else, or it
might involve writing our own API and polyfilling everybody. A
brief list of the things I can think of:</p>
<ul>
<li>FFI support (we'll need it for LDAP, OS integration, GSSAPI,
possibly things like S/MIME, import, PGP)</li>
</ul>
</blockquote>
<br>
TB:NG should be pure JS, with the exception of the OS integration
you list below. If some protocol implementation does not exist yet,
but it's possible, e.g. LDAP, we'll write it.<br>
<br>
For those things that absolutely do need native calls, because of
e.g. Kerberos credentials, we might still not come to the IPC
question at all, because there's already a JS wrapper on npm, e.g.
<a class="moz-txt-link-freetext" href="https://www.npmjs.com/package/kerberos">https://www.npmjs.com/package/kerberos</a><br>
<br>
Let's cross that bridge when we get there.<br>
<br>
<blockquote
cite="mid:5a31e3ae-e621-94df-5253-30d986ac674c@gmail.com"
type="cite">
<ul>
<li>Filesystem</li>
<li>Sockets<br>
</li>
<li>DNS<br>
</li>
</ul>
</blockquote>
<br>
This is all integral part of the node.js/npm platform, e.g.
require("fs").<br>
<br>
Remember that if we use existing npm libraries, they will already
presume certain npm dependencies. E.g. if we use emailjs.js IMAP
implementation, it already uses a certain socket module, likely the
native node.js one. We should look out that they are somewhat
consistent (not 13 different socket implementations by various
libraries), but I don't see a point in rewriting library code to use
our artificial wrapper.<br>
<br>
Essentially, there are already so many JS libraries on npm, and they
give so much value, that at this point, I'd make npm support a
requirement for any platform that we'll use. Including the platform
APIs they use, e.g. node.js require("fs") and sockets etc. That's
basically a requirement for all the already existing code.<br>
<br>
So, it's hard to make such decisions in isolation. We should first
get our feet wet and make a prototype, and gain some experience with
the new situation, and then we can derive coding standards and
platform APIs from there.<br>
<br>
Essentially, I don't want to re-do NSPR in JS. That was probably
Mozilla's biggest mistake, to write their own custom platform
abstraction. It was a necessity back then, and maybe there was
nothing better, but it led to the arcane platform it is today. Let's
try to stay in sync with the existing JS community, and use the same
platform APIs. That gives basis for collaboration, in both
directions.<br>
<br>
<blockquote
cite="mid:5a31e3ae-e621-94df-5253-30d986ac674c@gmail.com"
type="cite">
<ul>
<li>Database (I think we need ACID-compliant database, and I
don't think we want to roll our own here. Note too that "load
it all into memory" is not going to satisfy performance
requirements when dealing with large folders).<br>
</li>
</ul>
</blockquote>
<br>
Again, let's cross that bridge when we get there. For starters, I
just need preferences, mbox, and JSON. That gets me most of my way.<br>
<br>
In some other XUL project, we used sqlite, and we eventually
replaced it module by module with simpler solutions, and ended up
with no DB at all.<br>
<br>
<blockquote
cite="mid:5a31e3ae-e621-94df-5253-30d986ac674c@gmail.com"
type="cite">
<ul>
</ul>
<h2>Boring infrastructure stuff that can be surprisingly
controversial</h2>
<p>Where do you host repositories? Issue tracking? Roadmaps? Build
instructions? Support? Documentation? Continuous integration?
Mailing lists? Updates? What license are you going to use? What
build system? Test infrastructure? Documentation tool? Linting?
Static analysis? Some of this stuff can be deferred or changed
after coding begins in earnest. Some of it can't.</p>
</blockquote>
<br>
We should not even going to discuss this at this point, and just
delegate this to a person to "set up our servers". Having a
community discussion on this is controversial and a waste of time
for everybody, or "rathole" as Gerv rightly called it.<br>
<br>
Just one meta thing: We should not require contributors to make
accounts on third party servers, nor use any proprietary software.
(I'll make a single exception for github, because so much of the
community is already there, and even that could be replaced with
gitlab later.)<br>
<br>
Ben<br>
</body>
</html>