Proposal to start a new implementation of Thunderbird based on web technologies

Paul D. Fernhout pdfernhout at kurtz-fernhout.com
Tue Mar 28 02:46:00 UTC 2017


Hi Ben,

This is a great proposal! Some ideas below to build on it.

## Minimal Viable Product

I completely agree the complete conversion effort in total will take
years (especially to move the plugin ecosystem over). But, to maintain
significant interest in the effort, I'd suggest aspiring for having a
minimally usable system to view and send email as plain text within a
month of starting (as in, one person full-time for a month).

## Thunderbird interoperation from the start

To make such a "eat our own dogfood" application useable from the start,
consider having Node.js write to the same mailbox files, address book,
and filter specifications that regular Thunderbird uses. Then POP users
(like myself) can use the new app for what it does (plain text email),
while also always being able to fire up the new Thunderbird for what it
can't do yet (like some plugins for calendaring or encryption or doing
IMAP). We can make a lot of progress on a POP-only Thunderbird's UI
(plus RSS feeds etc.), and then at some point IMAP support could be added.

Yes, anyone who does the above is living dangerously as they put their
real email at risk and so needs to make frequent backups. In order to
make this safer, we could perhaps tweak the existing Thunderbird (as a
config option) to check a lock file (or whatever) before starting up so
the two apps will not both work at the same time on the same files and
confuse each other. Or maybe we could ideally figure out a smoother way
for both applications to both be running at the same time through
cooperative file locking or such. Or maybe to minimize the risk of
corruption we could make some less minor changes to the existing
Thunderbird codebase so it could serve up and receive messages,
addresses, and filters via a socket to a Node.js server which then
serves the web-technology UI (but that is starting to mean more
wrestling with the existing codebase which might introduce new bugs and
probably won't be as much fun).

## Mobile from the start

Also, I'd suggest adding an extra mobile aspect early so that you could,
if you want, check your Thunderbird email from any web-enabled device
like a smartphone, tablet, or Chromebook that can access your
Thunderbird backend server. That new feature could be a big win for home
or corporate users relying on POP. Obviously, security is an issue if
you let the new Thunderbird backend serve such remote apps.

## Analytic Visualizations from the start

Another thing to add to the new Thunderbird from the start is
visualization features and analytics, like showing you graphs of who you
email with over time about what. This could be done in readonly mode for
existing mboxes. A big value of this is to get people installing
this new web-technology Thunderbird for just these analytic features,
even if they do their actual emailing mainly entirely in the existing
Thunderbird at the start. In fact, I'd suggest we could even consider
start with this part (reading mbox files without writing them) as a
Thunderbird companion and then build out from there.

## Technology stack

I'd suggest using TypeScript, Mithril.js, Tachyons, and
Node.js for the base technologies -- probably with Mocha/Chai for
testing. Later we can make a more stand-alone desktop version with
Electron or such. We could use socket.io to push notifications for
changes in real or virtual folders and then use Mithril.js (a vdom
library) to quickly rerender changed data the client UI requests (or is
pushed). I could explain those choices further but it would take a while.

## A previous step forward by me

Here is my past attempt in the direction of a web-technology-based
Thunderbird Server (with some help by David Krings for design and
testing) -- but I ended up taking a full-time+ paying job doing UI work
for a SV company soon after starting that effort, so not much time for
moving it further along, sadly.
https://github.com/pdfernhout/Twirlip2

A key useful thing I developed there (improving on something first
developed for another project) was displaying HTML reasonably safely
using Mithril (crossing fingers).
https://github.com/pdfernhout/Twirlip2/blob/master/source/webapp/ts/sanitizeHTML.ts

I use that sanitizer in the only significantly working plugin there (an
RSS feed reader where you can choose your level of danger for displaying
HTML[1]):
https://github.com/pdfernhout/Twirlip2/blob/master/source/webapp/ts/plugins-for-local-libre-or-standard-services/rss/rss-plugin.ts

A demo of the RSS feed reader plugin can be found at the bottom of this
page (it's a test server and may not stay up indefinitely):
http://twirlip.com:9000/twirlip.html

## On grids and performance

On a tangent about grids since they are of concern, here is a grid in
Mithril.js that can display tens of thousands of items scrolling fluidly
in the browser using a vdom approach:
http://jsfiddle.net/7JNUy/1/

I discuss that grid scaling issue further here:
https://github.com/dojo/meta/issues/11#issuecomment-177547645

## Learning from my mistakes

One of the (many) too ambitious parts of the above effort was to use a
different storage backend. Starting using mbox in parallel with the
current Thunderbird (as above) would have been a better strategy at the
start. So would have been would have been focusing on visualizations of
those existing mailboxes.

## Future directions

Once all that works, and there is useable UI frontend, we can talk about
having plug-able backends perhaps to support advanced UI plugins (like I
brainstormed in the plugin directory names in that project).
https://github.com/pdfernhout/Twirlip2/tree/master/source/webapp/ts/plugins-for-local-libre-or-standard-services
https://github.com/pdfernhout/Twirlip2/tree/master/source/webapp/ts/plugins-for-specific-websites-or-proprietary-services

--Paul Fernhout (pdfernhout.net)
"The biggest challenge of the 21st century is the irony of technologies
of abundance in the hands of those still thinking in terms of scarcity."

[1] Ultimately the safest way to handle displaying full HTML might be to
have every email or RSS feed be displayed its own subdomain served by
the server using the item's message ID...

On 2017-03-24 1:04 PM, Ben Bucksch wrote:
> Summary:
> 
> * Our base goes away. Gecko will change dramatically in the future, 
> and dropping many features that Firefox does not need anymore, but 
> that Thunderbird relies on. * Our codebase is now roughly 20 years 
> old. It heavily and intrinsically relies on those very Gecko 
> technologies that are now being faded out, more or less aggressively.
> * JavaScript and HTML5 have evolved dramatically. Entire applications
> in JS are now realistic, and have been done. There are several
> existing JS libraries we might leverage. JavaScript is an efficient
> language, which allows fast development. A rewrite in JavaScript
> makes sense now. * We will learn from shortcomings of existing
> Thunderbird, and solve them, for example a more flexible address
> book, and cleanly supporting virtual folders without overhead. * The
> goal of the rewrite is to be close to the existing Thunderbird, in UI
> and features, as a drop-in replacement for end users, without
> baffling them. They should immediately recognize the replacement as
> the Thunderbird they love. It will install and run as normal desktop
> application, like Thunderbird does today. It keeps user data local
> and private. * We can also make a new, fresh desktop UI, as
> alternative to the traditional one, for new users. The technology
> also gives us the option to run it as mobile app. * While we
> implement the new version of Thunderbird, the old codebase based on
> Gecko will be maintained until the rewrite is ready to replace the 
> old one. * I expect this effort to take roughly 3 years: 1 year until
> some dogfood (usable by some developers and enthusiasts). 2 years
> until a basic feature set is there. 3 years until we can replace
> Thunderbird.
> 
> I will describe each point in more detail below.
> 
> 
> Proposal
> 
> I would like to put this up for discussion. I have not discussed this
> with the rest of the council yet, but rather I start the discussion
> here, to make it public, so that everyone can follow the discussion.
> 
> Ultimately, this will be a team effort. The TB Council, even with 
> employees, will not be able to do this alone. I hope that Thunderbird
> can fund a number of developers on this, but I would like this to be
> an open-source project in the sense that everybody can participate.
> Please see this also as a call for help. If you are an experienced
> JavaScript developer who already wrote applications and libraries in
> JavaScript, please step forward and write us that you want to help.
> 
> 
> Importance
> 
> I've been working on Thunderbird since 18 years, and I care about it 
> a lot. More importantly, I care about communicating with my friends 
> in a way that's private and not controlled by any single entity, but
> decentralized and running and stored on my own computer. In other 
> words, we need a desktop (or smartphone) email client. Not just 
> today, but also 30 years from now, if the world didn't collapse
> until then.
> 
> This is important for the freedom of the individual, of 
> communication, and of computing. There are not many contenders left 
> that defend the freedom of communication, because most have an 
> interest to bind users to their own services. If I want to 
> communicate with teenagers today, or even my mother, the best way is 
> WhatsApp or SnapChat, closed systems. In order to make a different in
> this world, Thunderbird needs to be a viable option, not just next 
> year, and not only for the existing userbase, but for the whole 
> world, all age groups and all usage patterns. The current
> Thunderbird codebase has served well - 25 million users are amazing -
> but will not be able to defend freedom for all of us. Given the
> current technical challenges we face, this is the perfect time for a
> fresh start.
> 
> 
> Our base goes away
> 
> Firefox has discontinued the Firefox XUL extensions, killing tens of
>  thousands of small projects this way. This is a massive cut that
> goes to the base of Firefox. They did that, because these extensions
> use XUL and XPCOM, and this is preventing Firefox developer from
> making more drastic changes in the internal architecture. They are
> gradually moving to using some Rust components, which means the XPCOM
> base is no longer viable. At the same time, they also want to
> gradually decrease reliance on XUL in favor of HTML, and remove parts
> of the XUL implementation that they no longer need to run Firefox.
> That's why they remove XUL extensions. Thunderbird heavily relies on
> XUL, all its UI is written in XUL. Some parts of XUL - esp. some
> features of the tree widget - were back then written only for
> Thunderbird. We can expect breakage in this area, as Mozilla has
> clearly stated that they will no longer care about breaking
> Thunderbird. In fact, I think this problem is part of the motivating
> force behind Mozilla asking us to be independent.
> 
> 
> Thunderbird relies on those very Gecko technologies
> 
> Thunderbird is based entirely on XPCOM (backend module API) and XUL 
> (user interface). If you take out XPCOM and XUL from Thunderbird, you
> have effectively rewritten it. This change is as dramatic as the 
> change from Netscape 4.5 to Mozilla was, when it was rewritten with 
> XUL and XPCOM. 90% of the code was new. We are faced with a similar 
> problem.
> 
> 
> Gradual change would be very costly
> 
> Of course, the changes in Gecko are gradual, but eventually, the 
> technology will go away, so the overall work is there nonetheless.
> 
> Given that XPCOM is the very technology that creates the API between 
> the modules, is hard to replace step by step. I think we would spent 
> as much time integrating the new modules with the old code as we 
> would take writing the module in the first place, which means the 
> overall effort increases at least 2-fold by trying to do it step by 
> step.
> 
> Having worked on Thunderbird since 18 years, I do not deem it 
> feasible to modernize it component by component. I estimate it would 
> be 4 times faster to start with a clean slate and start from 0.
> 
> 
> JavaScript is today the best choice
> 
> JavaScript, if used diligently and with good design, is a very 
> efficient language. Both in execution time, but more importantly for 
> developers. Personally, I wrote apps in many languages, including 
> C++, Java and JavaScript. Of those, JavaScript is by far the most 
> productive - I am personally 4-10 times as productive as with C++.
> 
> Today, we have node.js, Electrolysis, Cordova and other platforms 
> that are already ready-made to create desktop applications using 
> HTML5 and JavaScript. There are already other applications that have 
> been written this way, and there is existing knowledge and libraries 
> on how to do this.
> 
> 
> Solve shortcomings of Thunderbird
> 
> There are some known shortcomings of Thunderbird, as it is 
> implemented and designed today. Some are on the feature level, for 
> example the address book is not flexible enough to capture even 3 
> email addresses for one person, it cannot link 2 persons (e.g. a 
> couple), and so on. A modern re-write would consider today's user 
> requirements.
> 
> Deeper goes the problem of  virtual folders. Thunderbird has them 
> implemented as a bolt-on, which essentially keeps a copy of the 
> mails, as far as I understand, and they are slow and inflexible. A 
> new Thunderbird implementation could take an approach of where 
> folders are a view that is computed in real time (like views in 
> databases). This matches what GMail does with the AllMails folder
> and tags as properties, and then tags also form a virtual folder.
> There are important implications that this has: I can have a unified
> inbox, one inbox per folder, and I can have the emails sorted into
> specific project folders on arrival, all without overhead. The same
> incoming email appears in all 3 folders in the same second, and is
> marked read at the same time. I no longer have to move emails, but
> they are filtered.
> 
> Even deeper are design shortcomings of Thunderbird. It establishes a
>  rough border between user interface and backend modules for IMAP 
> etc., but is not very consistent in enforcing the independence of
> the 2 layers.
> 
> Right now, large parts of the logic are written as part of the 
> frontend code, which is one of the big reasons why Thunderbird code 
> is so difficult to understand and modify. It also makes it 
> practically impossible to write alternative frontend. Instead, the 
> architecture should be flexible enough to allow this, by strictly 
> separating logic from UI using design patterns.
> 
> Some backend modules call the frontend, to update changes. Instead, 
> we should use a classic observer/subscriber design pattern, where the
> frontend subscribes to changes in the backend, and updates itself.
> That allows more flexibility for the frontend. All lists should be
> observable, which is a simple but very powerful way to decouple logic
> from UI. See Linq in C#, and 
> https://wiki.mozilla.org/Jetpack/Collections , which takes it a step
>  further. This could be the technical basis for very flexible virtual
>  folders that update immediately and are fast. But it would be used 
> whenever there is a list.
> 
> 
> Be close to the existing Thunderbird
> 
> Even though we write almost all code from scratch, we will save a
> lot of time by having a clear goal: We want to replicate the current
>  Thunderbird, from an end user perspective. That means, the user will
>  find the same 3-pane window layout, the same way how folders and 
> message lists and the thread pane operate. The theme will be
> similar. Existing Thunderbird users should feel right at home.
> 
> We retain the overall UI and most features and qualities like 
> performance, even if we do not copy all little details.
> 
> Disclaimer: Given that the technological basis - particularly HTML - 
> is completely different, there will be some things that work 
> differently in some ways. Hopefully, many will be better. We will 
> have some technical limitations. Some will be just different,
> because the underlying implementation is completely different. The
> goal is not to copy bug for bug, but to make existing users
> immediately recognize this as Thunderbird, and feel at home, even if
> some details are different.
> 
> We must pay attention to also keep technical qualities that many of 
> our users rely on. An obvious one is that the new implementation
> must be able to quickly scroll through a list of up to 100000
> messages. There is no such HTML widget that allows that, we would
> have to create one, but I think it's feasible, I already have ideas
> how to do that. We also need to preserve privacy and security
> qualities that Thunderbird has, or even improve on it.
> 
> 
> Fresh UI, more platforms
> 
> In addition to replicating the current Thunderbird UI, we should also
> experiment with new forms of UI, in parallel. For example, we should
> create a UI that's suitable for the new generation of users that
> never used a desktop email client before. These people do not feel at
> home with Thunderbird today, and we should create something for
> them.
> 
> A lot of the new userbase is on mobile platforms and on tablets. That
> new UI should be "responsive" (automatically adapting to different
> screen sized) so that it runs well on tablets and smart phones as
> well. With Cordova and similar toolkits, we have a technological
> basis to quickly make a mobile app out it so that it installs like
> any other app. It would be a replacement for the system "email" app.
> 
> The goal for the new UI are 1 billion users.
> 
> 
> Maintain old Thunderbird code base
> 
> During the time while the new rewrite is implemented, a smaller part 
> of the staff will maintain Thunderbird, as they have done in the last
> 2 years. We keep up with Gecko changes, and fix smaller bugs. But we
> will not do major refactoring or big new features on the old 
> codebase.
> 
> This gives users a continuously updates Thunderbird. Particularly 
> important are security updates for security holes found in Gecko, 
> which Thunderbird inherits. They might be exposed in HTML emails,
> RSS feeds or other places where Thunderbird shows HTML and renders 
> images.
> 
> 
> Effort
> 
> The majority of the team should concentrate on the rewrite. This is 
> also why this decision is important to make now, because Thunderbird 
> has now resources to hire a number of developers, and we need to 
> decide on which efforts we spend the precious little money we have 
> through the generous donations of our existing users.
> 
> I would estimate that we need a team of 10 full time developers, for 
> 3 years. We need 2 persons for the framework, 3 for backend modules, 
> 4 for frontend UI, and 1 for theming.
> 
> 
> Year 1
> 
> In the first year, we are laying the foundation, getting the 
> framework sorted out, building infrastructure etc..
> 
> After 1 year, I would expect a first demo that can read and write 
> email, but with a very minimal feature set and many rough edges. A 
> few enthusiastic first alpha users might be able to use it for their 
> email needs, and some developers can use it for their daily needs.
> 
> 
> Year 2
> 
> In the second year, we would concentrate on building the important 
> features that are needed for most users.
> 
> After 2 years, we should have an email client that's appealing to 
> most normal users. Even some power users might like it, because we 
> have advanced features that no other client has, but they will miss 
> some features.
> 
> 
> Year 3
> 
> In the third year, we concentrate on feature parity to the old 
> Thunderbird. We add any features that Thunderbird currently has and 
> are appreciated by the existing userbase. We also add functionality 
> that allows larger deployments, as a significant part of userbase are
> enterprises.
> 
> Any features that are used by significantly less than 1% of the 
> userbase will not be implemented, in favor of other features that
> are desperately needed today.
> 
> After 3 years, 95% of the existing Thunderbird userbase should find 
> the features they need in the new implementation. There will be some 
> changes and adoptions necessary, but there should be a way to do what
> they need. That's what I call "feature parity".
> 
> 
> Year 4-20
> 
> Keep improving
> 
> 
> 
> 
> _______________________________________________ tb-planning mailing 
> list tb-planning at mozilla.org 
> https://mail.mozilla.org/listinfo/tb-planning
> 


More information about the tb-planning mailing list