What happened to hiring an architect?

Andrew Sutherland asutherland at asutherland.org
Sun Dec 11 07:45:49 UTC 2016


On Sat, Dec 10, 2016, at 01:28 PM, Ben Bucksch wrote:

>  Personally, I do think there are a number of questions to answer:

> 
>  * What runtime to work on - Electron, Cordova, or Gecko?



Firefox WebExtension.  I say Firefox rather than general WebExtension,
because the Firefox WebExtension team has indicated a desire to go
further than the APIs provided by Chrome, such as providing a real
sidebar extension point.


Benefits:
 * Access to the TCPSocket API.  While JMAP moots the need for TCP
   connections, it will probably take some time to change the world.
 * A "safer" distribution model than being a website.  The web trust
   model doesn't currently support the concept of signed code.  Although
   there have been some clever hacks involving key-pinning and app-
   cache, a pure web app depends on the server not being compromised
   (even if using ServiceWorkers).  The extension distribution/update
   model allows for strictly versioned releases that provides the user
   with more control over upgrading, etc.
 * Improved security for the user by having their mail app run with
   reduced permissions while also running on an evergreen, up-to-date
   browser.  As a WebExtension, an attacker exploiting a bug in the mail
   app is limited to the capabilities of the mail app, which are
   strictly less than those of the browser itself.
 * Along those lines, extensions to the mail app can also be structured
   as Firefox WebExtensions that have their own permissions (and
   origin/storage).  This means that something like enigmail can run in
   a completely separate execution context with completely separate
   storage with interaction limited to requests like "encrypt this blob"
   and "decrypt this blob".  Enigmail can also have its own UI which
   handles password entry and when it's willing to encrypt/decrypt.
   Similarly, an extension that wants to interact with web-pages using a
   content script to overlay information/extract data not available via
   other protocol/etc. can have a tightly scoped content script
   permission.  Again, this limits the worst-case scenario in the event
   the core mail app ends up having a security bug.
Corresponding downsides:
 * WebExtensions are constrained to the APIs available to them.  I'm not
   aware at this time of any functionality a mail/calendar client would
   need that would not be available, but I can believe there may come a
   time where the limit is hit.  The good news is that
   https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Native_messaging
   provides somewhat of an escape hatch.

>  * ow do the backend modules communicate with the UI?



postMessage.  The GELAM (gaia email libs and more:
https://github.com/asutherland/gaia-email-libs-and-more) architecture is
that the back-end lives in a Worker that communicates with any UI pages
via postMessage/MessagePorts.  Although it may be conceptually appealing
to expose the contents as a REST API via ServiceWorker or similar, there
are significant benefits to leveraging postMessage's use of the
structured clone algorithm because it allows us to pass Blobs/Files
around as efficient references to content stored on disk (or in memory).

>  * What are the main APIs? High-level overview.



UI:



For the general browsing experience, clients ask the back-end to
subscribe to ordered list views or specific items.  At
https://github.com/asutherland/gaia-email-libs-and-more/blob/10831e8706542f44043d5e180534f1e1a845b9f5/js/mailapi.js#L885
you'll see viewAccounts(), viewFolders(),
viewFolderConversations(folder), viewConversationMessages(conv) plus
searching/filtering variants.


For the potentially big N cases of conversations and messages in
conversations, this subscription is to a windowed subset of the entire
ordered list, with that window characterized by "buffer above",
"visible", and "buffer below" to help the back-end prioritize its
actions such as fetching conversation snippets, etc.  The objects in the
lists are live-updating.  Live-updating object references that are
independent of the list window can also be obtained (ex:
getConversation(id), getMessage(id) from
https://github.com/asutherland/gaia-email-libs-and-more/blob/10831e8706542f44043d5e180534f1e1a845b9f5/js/mailapi.js#L317).


Manipulation of messages is all a unidirectional flow like you'd find in
the react ecosystem of flux/redux/etc.  You issue a request to
manipulate conversation(s)/message(s) to the back-end; it does the
manipulation of local database state, which cascades through the
subscriptions and then your event listeners on the objects or their
owning list are fired and you can update your UI.


Backend overview:



There's a "task" infrastructure.  There's more details at
https://github.com/asutherland/gaia-email-libs-and-more/blob/10831e8706542f44043d5e180534f1e1a845b9f5/js/task_infra/tasks.md
and in the jsdocs on the classes in that directory.  The high level idea
is that all manipulations of database state must happen within a "task".
Tasks are atomic and ideally as small as they can possibly be.  There's
a "planning" phase when fast local mutations are made ASAP (like
speculatively starring a message, moving a message between folders) and
"execution" when online operations or heavy processing run.  IndexedDB
transactions, the data layer being clever, and responsible tasks which
speculatively commit "undo" tasks when issuing dangerous or non-
idempotent requests to servers are used to keep the database coherent
and able to efficiently recover in the event of crash.


Planned-but-not-yet-implemented, tasks are structured to be parallelized
with explicit resource dependencies and read and write-locks on the
database so that operations can run in parallel as long as they do not
interfere with each other.

>  * Do we support extensions, and if so, which kind of API do we offer
>    them?


Absolutely.  For my thinking, the primary questions are:
 1. Security: How do we let users easily try out extensions without
    putting them at risk of security/privacy problems?
 2. Developer-friendly: This is a bi-directional thing; how do we make
    it easy for developers to implement extensions and for other
    developers to review those extensions?
 3. Growing: How do we help grow the Thunderbird developer community?


The answer I'm coming up with is that we want to:
 * Create very explicit extension points.
 * Build all "core" functionality on extension points to the maximum
   extent possible.
 * Add more explicit extension points as would-be extension developers
   put forth an idea that current extension points are insufficient for.
 * Work with the developer to put their extension in the core as a
   default or an option, hopefully interesting them in writing more
   extensions and/or helping be part of the "core" development.  (Which
   is to say, all of it, not just the extension they contributed, but
   that's good too.)


I started down this road when I implemented the faceting support I
described in my blog post at
https://www.visophyte.org/blog/2016/03/21/web-worker-assisted-email-visualizations-using-vega/.
The faceting implementation exposes its endpoints at
https://github.com/asutherland/gaia-email-libs-and-more/blob/10831e8706542f44043d5e180534f1e1a845b9f5/js/extras/vis_facet/worker_def.js
and is explicitly "activated" by the glodastrophe static configuration
(which could be user-configurable in preferences) at
https://github.com/asutherland/glodastrophe/blob/status/www/app/felam/worker_extensions.js
and in the UI containers like
https://github.com/asutherland/glodastrophe/blob/status/www/app/containers/selected_overview_facets_pane.js
are responsible for rendering the (currently hardcoded) schemas
configured at
https://github.com/asutherland/glodastrophe/blob/b8851ef56978e588a7ad9c298ea50d9d17e79f79/www/app/reducers/viewing.js#L90
using the widgets in
https://github.com/asutherland/glodastrophe/tree/b8851ef56978e588a7ad9c298ea50d9d17e79f79/www/app/components/visualizations.


More directly answering your question, this means that the set of
extension APIs starts small and grows as people come up with extensions
they want.  Although I discuss a scenario of "peer" WebExtensions that
interact with the backend, I would expect those to come later.  I would
expect starting with simple equally-privileged extensions in the backend
and possibly some less privileged userscripts-style mechanism where JS
code is executed in an isolated, unprivileged sandbox where it is given
very well-defined inputs and it returns very well-defined outputs and
has no access to the network, etc.


Note that I'm largely talking about back-end extensions here.  See the
next answer for UI-related discussion.

>  * What target platforms - desktop or also mobile?
>  * What UI?



Because UIs use the "MailAPI" as clients, it's possible to have multiple
UI's, some of which live in their own WebExtensions.  With modern
application construction using react and redux, it's also possible to
build up a modular library of widgets that can be reused and replaced
with alternate visions of how the UI should work.


Fundamentally, I think desktop and mobile probably need to be different
UI's, but can hopefully reuse many of the same widgets.  For the
"glodastrophe" prototype, I was going explicitly for desktop, but with
the material UI widgets reconfigured with touch padding, a lot of reuse
is possible.


Touching on privacy again, I think UI's are an area where there could be
a class of webpages or other webextensions where the data being provided
to the alternate UI is made very explicit.  Perhaps a "share" approach
where you explicitly share the contents of a folder, or a conversation,
or a search result/etc. with the alternate UI.  Or a pull/request
approach along the lines of OAuth where you grant an alternate UI access
explicitly based on its request which amounts to a
folder/conversation/search/whatever.

>  * Do we maintain a folder-centric UI, or hierarchical labels, or
>    dynamical folders based on gloda?


This is an area where the (post-)conversations GELAM implementation
really excels.  On disk (in IndexedDB), messages are clustered by
conversation to provide locality.  The ordered lists I discuss above are
implemented as what amounts to arbitrary materialized views with enough
information for the virtual list to efficiently seek and provide any
information that informs the effective coordinate space of the virtual
list.  For example, one iteration of the glodastrophe UI would use
between 1 and 4 rows per conversation, based on the number of unread
messages in the conversation.  This was baked into the "TOC" (Table Of
Contents) so that the virtual list could seek perfectly as the user
dragged without loading any of the conversation objects into memory.


(Also nice in that example, the summaries of the up-to-3-unread-messages
get summarized into the conversation data-structure, so as you scroll
through the conversation list, we're never having to load messages, just
conversation summary objects.)


Getting back to your question, as long as you can deterministically-in-isolation-from-its-
neighbors determine what ordered lists a
conversation/message/whatever belongs to and what, if any,
immediately available meta-info is required,  you can view things
that way.  It's really up to what the UI wants.




> OK. Sounds good! What's stopping you? 



https://www.youtube.com/watch?v=3wxyN3z9PL4



Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/tb-planning/attachments/20161211/ff5f8b45/attachment-0001.html>


More information about the tb-planning mailing list