Bundling vs sending serialized dependency graph

Ian Hickson ian at hixie.ch
Thu Aug 21 13:55:15 PDT 2014

On Thu, 21 Aug 2014, John Barton wrote:
> I think your graph is upside down from mine ;-) As I learned it, leaf 
> nodes were the ones at the ends of branches and hence were not dependent 
> on any other nodes; no node depended on a root node.

I don't really mind which way we view the graph. To put it your way: I 
mean a world where different otherwise unrelated root modules or resources 
depend on common shared dependencies.

> > What exactly is needed depends on what posts are displayed, the user's 
> > preferences with respect to features like Hangouts, etc. It's a 
> > complicated graph.
> The only issue that matters for the efficiency of bundle loading is how 
> many nodes are shared between commonly used dynamically loaded root 
> modules. If the module is needed always it will be loaded always. If the 
> module is only used by a single dynamically loaded feature, then there 
> is no penalty for bundle loading. Even in the case where two or more 
> dynamic loads use the same large number of modules we can simply put 
> those modules in a shared bundle. So the case where bundles lose is very 
> rare.

I think bundles are clearly always a win, since at the extreme (one 
resource per bundle) they just boil down to the same as no bundles. 
They're never worse than not bundling.

But they don't solve the dependency problem.

> Here is an attempt and a graph:
>   A   B  C  D   E
>   | /  \ | / \ /
>   R      X    Y
> R is always loaded so its bundle loads A and B.
> Optional feature X loads C and D but not B its already loaded.
> Optional feature Y loads E and possibly DE if X did not already load.
> Only D is extra work and thus this scenario is relatively rare.

Rare in this case. In practice, large sites have thousands of modules with 
many layers of depth in the graph. Modules just change the scale of the 
problem, they don't remove the problem.

> Typically optional modules will share dependencies with the default 
> loaded page modules or have unique modules.

Can you elaborate on how you determine that this is the typical case?

Consider G+, or Facebook, or other sites of that nature, where there's 
hundreds of "leaf" modules (different dialogs, post types, views, etc), 
and there's modules for each widget (search toolbar, app drawer, 
notification bell, share box, nav menu, tab strip, chat widget, post box, 
post box photo view, photo picker, ACL entry box, photo uploader, button, 
checkbox, radio button, link picker, video picker, event picker, theme 
picker, drop down button, date picker, time picker, link widget, time zone 
picker, location picker, map, multiline text box, people browser, list 
picker, hovercard, the list goes on and on and I've barely scratched the 
surface of what G+ has here), each of which depends on a number of 
submodules and so on. I don't think that having multiple levels of 
dependency is rare at all.

> > > But sure, it would be great to have a complete solution if it's not 
> > > a lot more complex.
> >
> > I don't think it should be particularly complex. It only requires some 
> > minor changes. One is that we need to be able to declare dependencies 
> > ahead of the "instantiate" hook.
> By the way I recently discovered that the deplist returned by the 
> instantiate hook does not enter the dependency graph analysis. These 
> aren't dependencies in a list rather a list of things needed to be 
> loaded.

I'm not sure what you mean here. The list returned from "instantiate" is 
treated the exact same way as the list auto-discovered from "import" 
statements when "instantiate" returns undefined: it's passed to 
ProcessLoadDependencies(), which calls RequestLoad() and 
AddDependencyLoad(), which, if necessary, updates [[Dependencies]].

That's all I'm talking about. I want to be able to update [[Dependencies]] 
earlier than "instantiate", and I want to be able to mutate 
[[Dependencies]] to remove nodes that are no longer dependencies before 
the load is complete.

> > Another (a subset, really) is that we need to be able to declare 
> > dependencies for ES6 modules as well as letting the ES6 infrastructure 
> > discover them automatically.
> In theory this should be straight-forward. In practice, well good luck.

Good luck with what?

> > Finally, it would be ideal if we could also adjust those dependencies 
> > on the fly, since if we're reflecting dependencies described in the 
> > mutable DOM structure, it might be mutated.
> I think this one is technically difficult.

I don't think anyone here is going to shy away from technically difficult 
problems, it's kind of our bailiwick. :-)

The idea is, in fact, to move as many of the technically difficult 
problems from things authors have to keep reinventing to things that 
browsers just support natively.

> > > I guess you're proposing the send the dependency graph to the 
> > > browser, then when a new root is needed, the stored graph is 
> > > compared with the currently-loaded modules. The additional modules 
> > > needed are then requested as a group. Up to this point we can just 
> > > use build tools and browsers.
> >
> > Actually, modulo the changes described above, the ES6 loader already 
> > does all this.
> Huh? How do you plan to parse the modules to obtain dependencies without 
> sending them to the browser?

You send them to the browser, just not in the module itself.

> > It just doesn't quite handle it at the level of pre-emptive 
> > declaration of dependencies. But suppose you had two modules A, B, and 
> > C. A and B depend on C. With ES6 today, when A is loaded, it loads C. 
> > If late you load B, B doesn't reload C; it just links into it. So this 
> > is all already supported. All that's needed is a way to tell the ES6 
> > system to get C before it has even received A.
> You've really lost me now. I thought your goal was to avoid sending C 
> over the network. Now you want to send it without even seeing A?

Not sending C over the network at all wouldn't work, since it would mean A 
doesn't have its dependencies available. I don't follow.

The idea is that authors be able to predeclare (relevant parts of) the 
dependency tree such that when a node in that tree is needed, e.g. A in 
the example above, all the relevant nodes can be fetched in parallel, e.g. 
A and C in the example above, rather than in a serialised manner, e.g. 
first fetching A, then parsing it, then fetching C.

One way to do this would be to predeclare the modules, as in:

   <script type=module src=a.js id=a needs=c load-policy=when-needed>
   <script type=module src=b.js id=b needs=c load-policy=when-needed>
   <script type=module src=c.js id=c load-policy=when-needed>


   // in some script or event handler or some such
   document.scripts.a.load(); // a is now needed, a and c get fetched in 
                              // parallel with only one RTT.

Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

More information about the es-discuss mailing list