Week-long sprint to create minimal usable Thunderbird Server (week one update)

Paul D. Fernhout pdfernhout at kurtz-fernhout.com
Sat Dec 19 05:18:04 UTC 2015


The sprint for a Thunderbird webapp & server proof-of-concept is moving 
along -- more slowly than I had hoped, although that was not unexpected. 
I knew each "day" I had listed should really be a week if I was being 
realistic and said that in the manifesto -- but if I was realistic I 
would not have started. :-)

Related: :-)
http://www.freesoftwaremagazine.com/articles/free_matter_economy?page=0%2C1
"Real research is full of unexpected set-backs and cost-overruns. Real 
researchers are full of optimism and unrealistic deadlines. The skills 
for research, development, logistics, and management are rarely found 
all in one person — good scientists rarely make good accountants, let 
alone good receptionists."

Some more detailed notes on how things are progressing are below. 
Hopefully I'll have more progress to report next week. We'll see.

I even more now really shouldn't be doing this. :-) As Korben Dallas 
(Bruce Willis) said in The Fifth Element: "This is so stupid!" :-)
"The Fifth Element Police Chase"
https://www.youtube.com/watch?v=m38QcCyo110&t=13

--Paul Fernhout

== Learning

A bunch of time went to learning (sort-of market research), including 
evaluating open source server-based stuff like mailpile, Kolab, and 
Citadel, and watching an hour long video about Kolab and its community 
especially.
"Kolab Presentation at #FISL15" by Torsten Grote of Kolab Systems
https://www.youtube.com/watch?v=pVU0y-SKXnE

Incidentally, Torsten mentioned Thunderbird is not improving and plugged 
KDE's Kontact, sigh.

I also looked at discussions about some other proprietary systems, 
including discussions of Skype vs. Slack. Ideally I would have written 
up notes on that learning, but I did not think to do that. I've just 
been accumulating bookmarks. I was trying to address comments pointed 
out about maybe joining forces with such projects, to try to see where 
they are and where they are heading.

Hopefully I may write something up in the future on that. But in short:

* Citadel is in C, which is not where I want to go. But it's got a good 
community and longevity. It uses Berkeley DB as the primary data store. 
I looked at some of the webapp code, but did not see anything especially 
I might want to reuse from casual poking around (but I have a high bar 
to reuse as I like Mithril).
http://www.citadel.org/doku.php/installation:source

* mailpile is in Python, and looks really cool as far as the UI. It 
focuses on just email and encryption (but is thinking about plugins it 
seems). I can wonder if that GUI could be reused? However, it seems 
mostly templates processed in Python? Maybe with some fancy bits of 
JavaScript here and there? I know Smári McCarthy from another context 
(Open Manufacturing) and he's a great guy. It stores email in files and 
has a memory index. Wikipedia says they crowdfunded $163,192 for it.
https://github.com/mailpile/Mailpile
https://www.mailpile.is/blog/2013-08-15_Digging_Through_the_Details.html

* Kolab was especially interesting though as they generalize IMAP into a 
general data store and are truly committed to open sourcing everything 
they do (so, no Enterprise holdback). It has been funded mainly 
initially by a German government organization. It also has been funded 
by people who it consults for. Most recently, a related $103,541USD was 
raised to improve the interface of Roundcube which Kolab uses (usually) 
as its email client; I'm not 100% clear about the people involved in 
that funding effort, but someone told me they are connected with Kolab. 
The related company might have the biggest hope of being the next 
Automattic/WordPress perhaps? However, Kolab itself which integrates 
multiple services seems like a problematical thing for a single user to 
manage and maintain on a desktop. Kolab is also only for Linux. I guess 
Kolab could set that up as a VM like in Virtual Box though as an 
appliance. That difficulty in setup for any FOSS application suite can 
also be a great source of revenue for hand-holding (a perennial conflict 
when writing FOSS under a consulting-based funding model). Still, 
Automattic/WordPress has succeeded even with a "famous five minute 
install" and now easy upgrades, so it is not impossible -- but 
Automattic's funding model seems based more on hosting or support 
related to hosted accounts. So, the VirtualBox appliance approach might 
be the easiest way forward for an easy install. I have not had time to 
look through the source code there, but it seems to be some mix of PHP, 
Python, and Erlang?  Plus whatever else is in supporting modules (C++?). 
As with mailpile, I wonder if that webapp code could be reused? I may 
contact Kolab when/if Twirlip looks presentable to see if we could work 
together somehow though because I like their company's style of open 
sourcing everything -- see, now I can justify some of the time spent on 
this to my wife as "job hunting". :-)
https://en.wikipedia.org/wiki/Kolab
https://www.indiegogo.com/projects/roundcube-next--2#/
https://kolab.org/get-kolab
https://git.kolab.org/diffusion/

* I had a digression into looking at Sakai and the Apereo Foundation, 
because online learning management overlaps with a lot of these 
email/workgroup systems. That does make me wonder if an improved 
Thunderbird could become part of the educational scene somehow, at least 
the informal educational one?

* I did a search on "Skype versus Slack" and found a lot of interesting 
discussion that helps to understand a certain instant messaging space. 
Skype has many new features (we're mostly on old Macs on Snow Leopard 
and so can't upgrade Skype though). Since people's thoughts on Skype 
were formed from older versions, Slack has a mindshare advantage 
starting from having addressed these limitation and seeming fresher. 
Slack seems to be winning based on a focus on "teams" and also "company 
wide" integration. But both have their strengths and weaknesses for 
various situations once you've bought into their premises.

* A useful reference for seeing what else is out there as far as market 
research:
http://alternativeto.net/software/kolab/
http://alternativeto.net/software/mozilla-thunderbird/

== Progress report for the first week

Here is my progress for the first week (as above, not as much as I hoped 
sadly):

* There are 101 commits on GitHub at the moment. I tend to commit small 
increments, so that doesn't say too much. I tagged progress through now 
with "rss_feed_display".
https://github.com/pdfernhout/Twirlip2/commits/rss_feed_display

* The current Thunderbird Server / Twirlip code is reading (but not yet 
saving) RSS feeds obtained through the server (proxying to get around 
same origin issues) and displaying them with sanitization with 
Mithril/TypeScript in the webapp. I first adapted sanitization code from 
NarraFirma, started improving it, and then realized I could use the same 
approach I used to parse RSS feeds with DOMParser to parse the HTML 
using a whitelisting approach to build safe Mithril structures. I also 
found a nifty-looking more comprehensive sanitizer as wall called 
DOMPurify and hooked that in as well.
https://github.com/pdfernhout/Twirlip2/blob/rss_feed_display/webapp/source/plugins-for-local-libre-or-standard-services/rss/rss-plugin.ts
https://github.com/pdfernhout/Twirlip2/blob/rss_feed_display/webapp/source/sanitizeHTML.ts
https://github.com/pdfernhout/Twirlip2/blob/rss_feed_display/test/sanitizeHTMLTest.js

* Made about five different HTML sanitization options to choose from 
when viewing RSS content, given the feeds have HTML but are not from 
same origin. Viewing such content as safely as possible is a critical 
aspect of security for a Thunderbird webapp. One display option shows 
images while discarding likely tracking pixels. I probably should have 
not spent so much time on that, but it seemed crucial to having really 
usable code. Plus, I may be able to bring some of that back as an 
improvement to NarraFirma to help justify this time to my wife.

* Setup a Mocha test infrastructure. I had not used Mocha before, and it 
is nice, but another learning curve. Now I want to use it everywhere. 
:-) No CI yet though, and probably not anytime soon.

* Implemented a simple basic memory storage system for initial testing, 
but it is not indexing yet. It has Mocha unit tests though. :-) Not much 
to see in storage yet, and that class itself may change a lot. It is 
mostly just a stub for further growth.
https://github.com/pdfernhout/Twirlip2/blob/rss_feed_display/server/pointrel20151212/StoreMemory.js
https://github.com/pdfernhout/Twirlip2/blob/rss_feed_display/test/StoreMemoryTest.js

* Spent a lot of time thinking about semantic/triple-ish data storage 
architecture which I like to do, and writing up some related notes (in 
the project's design folder) including about possible using a "sharding" 
design to support sychronizing including through git.
https://github.com/pdfernhout/Twirlip2/tree/rss_feed_display/design

* It now seems clear that any desktop email client software will in the 
near future be managing data on the order of 100s of GB or more. For 
example, my own Thunderbird email store is about 15GB (just checking it 
now -- mostly from mailing lists I'm on). I though that might be big, 
but someone else posted on the tb-planning list recently that their 
Thunderbird store was about 50 GB. So, realistically, and expecting some 
growth, to avoid any near-future performance surprises, any new 
email-etc app needs to be designed to support on the order of terabytes 
of information and billions of messages all stored locally. That puts 
some constraints on the architecture (both in storage architecture and 
in user interface).

* While I'm likely starting with flat files (as much as I'm say a 
PostgreSQL fan and a CouchDB fan), I've realized if I want to use Apache 
Accumulo (or PostgreSQL, or CouchDB, or even MongoDB) at some point as a 
pluggable backend, I can just use such a service through Node.js, and 
have it running on the same machine or same network (maybe with 
VirtualBox appliance like I suggested above for Kolab). Why Accumulo 
specifically though? Multiple reasons, but one key on is that, if we've 
already paid the NSA billions to snoop on all our email (meanwhile 
ignoring real threats to security like leaving the US Office of 
Personnel Management unsecured), why not use some of the technology the 
NSA created to archive and search email to help us all archive and 
search our own email and other content privately? :-) I'm big on 
dual-use and swords-to-plowshares, given so much funding goes into 
"defense" anyway (an unfortunate situation, given US social and 
political dynamics). Here is another thing I wrote about that the other 
day (another distraction from coding, sorry) mentioning this project at 
the end and others as examples of the sorts of things computer 
scientists could be researching with all the defense dollars they are 
taking in:
http://ma.tt/2015/12/moral-responsibility-of-technologists/#comment-585177

* Made a plugin wish list of modules that it would be neat to have 
someday and checked that in (like RSS, IRC, email, files, git, IBIT, 
issues, mbox-import, memex, notes, and more). Here are the standard 
wishlist ones (there are also site-specific wishlist ones in another 
folder like for GitHub):
https://github.com/pdfernhout/Twirlip2/tree/rss_feed_display/webapp/source/plugins-for-local-libre-or-standard-services

* The 2011 Macbook Pro I do most of my development work on these days 
(did most of NarraFirma on it) developed a technical issue (black 
screen) that renders it unusable and I could not fix via various 
workarounds (it's been having display issue for over a year). So, I had 
to migrate to another machine (Windows). :-( Fortunately, I also have an 
older 2008 Mac Desktop in my office too (where I have Thunderbird). It 
sometimes freezes every few weeks, but so far, it's still coming back 
up. Isn't there some movie where the competitors start out in airplanes 
and as one thing breaks after another they end up racing on cars, then 
horses, and then eventually tricycles at the end? :-) I better get my 
Raspberry Pi, Arduino, Commodore 64, even perhaps New Micros embedded 
Forth computers ready for action. :-) After all, just look what Korben 
Dallas could make some old cab do -- although he might have upgraded it 
a bit. :-)

* I wrote a long email reply to Andrew on an issue he raised on this 
list related to web server vs. desktop, but I decided not to send it 
until I can send it from the new software. :-)

* I sent a few recruitment emails including to the Mithril mailing list 
-- no success, but some useful ideas. I also found out Richard Stallman 
seems to really value Thunderbird and Enigmail exactly as they are. :-) 
I wonder if it would be productive for someone from the Thunderbird 
Council to want to contact the FSF to talk about brainstorming funding 
Thunderbird-as-it-is perhaps in partnership with the FSF? Thunderbird is 
one of the easiest ways to use GnuPG, and the FSF cares a lot about 
that, especially the ease-of-use aspect. Just keep your emails to RMS 
down to about six lines each. :-)

Sadly, I've missed my ambitious Thursday (yesterday) goal to post 
something to the tb-planning list from the software. I'm now hoping for 
doing that sometime next week. So, this email was still sent from 
Thunderbird.

== Commentary

I've started down this "so stupid" path -- but with a smile on my face, 
like the fictional Korben Dallas who relished the chance to put his 
training and experience to good use. Having started writing code for 
pulling in information from RSS feeds (a warmup to IRC and email), as 
with one comment in the previous email about today's Slashdot article on 
the superiority of "Gopher", :-) my perspective has suddenly shifted 
when I'm looking at most websites, or at jobs to help companies make 
websites (yeah, still doing that for a "break"), or at funny commercials 
about internet services. Especially now, and related to what I said in 
my second Mozilla Governance email, I keep seeing how, compared to 
email, so many people are all trying to lock other people into golden 
shackles so they can make a profit, rather that just supply the content 
in an easy to use open standard (more like RSS, although even that can 
be messed up) with indexing and meta-navigation done separately in 
standard ways with FOSS to provide them keys to their own freedom. 
Getting more people to make that perspective shift might be key to 
building funding support long-term for a lot of full-time work on 
improving peer-to-peer communications.

The other thing I'm finding is that with something like the Node.js 
ecosystem, and also lots of existing libraries in JavaScript now (thanks 
in large part to part of Mozilla's success at improving JavaScript that 
many Mozillians can take pride in), given so many existing libraries, 
probably the amount of code that one might want to reuse directly from 
Thunderbird itself for a webapp (outside of email routines, which I have 
not gotten to evaluate yet) is probably approaching zero. :-) However, 
the knowledge in the Thunderbird community about distributed 
communications and user interfaces for them (based on a decade of 
practical knowledge) could no doubt really make Node.js email handling 
and other communication into something comprehensive, reliable, and easy 
to use.

I wrote another page worth of stuff here about reasons to do this, and 
then another page about finding funding for Thunderbird development 
related to government and foundations (funding for Thunderbird both as 
it is and as it might be), but, along with the reply to Andrew, I'll 
wait to indulge myself by publishing such stuff if at all after I get 
more coding done and can use Twirlip / Thunderbird Server to send it or 
post it elsewhere. :-)

Back to "so stupid" coding towards a proof-of-concept... :-) Or maybe 
sleeping first and then coding. :-)

--Paul Fernhout
http://www.pdfernhout.net/
====
The biggest challenge of the 21st century is the irony of technologies 
of abundance in the hands of those still thinking in terms of scarcity.

On 12/12/15 10:59 PM, Paul D. Fernhout wrote:
> Below is a link to an ambitious "hard fun" agenda I am pursuing over the
> next week with user stories and tasks to create a proof-of-concept for a
> Thunderbird Server application that runs locally and has a single-page
> webapp frontend in Mithril. The hope is that by the end of the week a
> new web-server-based system will work well enough to actually be useable
> directly (if no doubt painfully) by a development team to discuss
> further improvements of that software in a bootstrapping (Engelbart) way
> towards becoming a "social semantic desktop". Goals include reading and
> annotating RSS feeds, IRC chat messages, and Thunderbird and pipermail
> archives; receiving POP3 email and sending email via SMTP; and
> supporting creating and editing wiki pages and IBIS diagrams via sending
> structured emails composed by plugins. If the sprint goes according to
> plan (and things rarely do) hopefully this next Thursday I will be able
> to send an email to this tb-planning list from the resulting
> application. :-)
>
> As I said, it is an ambitious agenda, but it is not quite as impossible
> as it may seem at first given I've just spent a year making a single
> page FOSS webapp and have also written some other code that I can draw
> from. Of course the UX is going to be simple and ugly, but hopefully it
> will prove the concept anyway.
>
> So, at the end of the "ThunderbirdS are grow!" manifesto is now a
> week-long sprint plan, initial architectural choices, some of my
> motivations for doing the proof-of-concept, a long-term invitation to
> collaborate down the road if this first sprint produces some good
> results, an explanation of why the plan is broad but shallow, and some
> other contextual information -- including at least one very good reason
> I should not be doing this proof of concept. :-)
>
> Here again is a link to that document:
> http://www.pdfernhout.net/thunderbirds-are-grow-manifesto.html
>
> BTW, Just above that sprint plan is the contents of another email I sent
> early today to the Mozilla governance list and which is still pending in
> the moderation queue. In it, I suggest how I feel there is a fundamental
> confusion in the Mozilla mission statement between supporting a whole
> ("the internet", which includes email and peer-to-peer) and a part ("the
> web", which tends to be more centralized).
>
> Because this is not an official Mozilla project, I'm calling it
> "Twirlip" and not "Thunderbird" (not wanting to get into legal trouble
> with Mozilla for trademark infringement or to cause any confusion with
> Thunderbird Desktop users). That could obviously change down the road.
>
> Even if this specific project accomplishes this ambitious agenda in a
> week, the project may not go anywhere after that limited success though
> for all sorts of reasons. But even a limited success could still be a
> useful example for this tb-planning group to think about in terms of
> what a real Thunderbird Server project might look like if Mozilla (or
> someone else) got behind some future effort eventually. So, I hope this
> project can add to the discussion here one way or another.
>
> I created a GitHub project to use for that sprint. Not much to see there
> yet beyond the MPL 2 license file and the ambitious one week sprint
> agenda though, but here is the link:
> https://github.com/pdfernhout/Twirlip2
>
> If I don't respond to any emails about this project anytime soon, it
> will hopefully be because I am busy coding. :-) As above, I hope to
> provide an update on this project at the end of this coming week.
>
> Wish me luck! ThunderbirdS are grow! :-)
>
> --Paul Fernhout
> http://www.pdfernhout.net/
> ====
> The biggest challenge of the 21st century is the irony of technologies
> of abundance in the hands of those still thinking in terms of scarcity.
> _______________________________________________
> tb-planning mailing list
> tb-planning at mozilla.org
> https://mail.mozilla.org/listinfo/tb-planning
>


More information about the tb-planning mailing list