ModuleDeclarationInstantiation behaviour after failure

[Jon Coppeard]

On 15/07/2016 18:36, Allen Wirfs-Brock wrote:

> Can you point me at the parts of the HTML spec. that involved here?

The script tag is defined here:

https://html.spec.whatwg.org/multipage/scripting.html#the-script-element

The mechanism for loading modules starts here:

https://html.spec.whatwg.org/multipage/webappapis.html#fetch-a-module-script-tree

And HostResolveImportedModule is defined here:

https://html.spec.whatwg.org/multipage/webappapis.html#integration-with-the-javascript-module-system

> ModuleDeclarationInstantiation is a semantics of the ES spec. and it
> isn’t clear to me why the HTML spec. would need to invoke it.

The HTML spec needs some way of linking and executing modules.  It's not
clear (to me) which parts of ES can be "called into" and referred to
externally from another spec.

I guess at this point HTML is providing the source texts required by
RunJobs step 2:

"In an implementation dependent manner, obtain the ECMAScript source
texts ... for zero or more ECMAScript scripts and/or ECMAScript modules"

I'm still not sure how I would refer to that exactly from an external spec.

Further to this, if we do pre-instantiation of modules then we need some
way of making this happen before TopLevelModuleEvaluationJob runs.  I
don't think saying something like "use an implementation specific way to
pre-instantiate a module" would be that useful, whereas saying "call
ModuleDeclarationInstantiation for the module record" makes it clear
what is supposed to happen.

> (BTW, a loader API that includes the ability of ES code to
> programmatically introduce additional modules will also complicate eager
> or parallel processing.  If that is possible, then failure of eager
> processing of some module trees might be a temporary condition, and
> failure means that the processing of the module tree needs to be
> deferred until the API causes something to change.    In other words,
> statically fetched modules that have dependencies upon dynamically
> created modules are tricky )

I totally agree and thankfully we are not doing that yet!

> My sense is that you are seeing requirements/restrictions in the ES
> spec. that aren't there (or aren’t intended if they are).

I think you're right.  I'm not sure exactly what I had in mind but it
seems the answer is "do anything you like as long as it is not
observable to script", which is a lower bar than I realised.

Thanks,

Jon

On 15/07/2016 18:36, Allen Wirfs-Brock wrote:
> 
>> On Jul 15, 2016, at 7:32 AM, Jon Coppeard <jcoppeard at mozilla.com
>> <mailto:jcoppeard at mozilla.com>> wrote:
>>
>> Thank you for the detailed response.
>>
>> To start with I should say that although I'm working on an
>> implementation of the basic module loader defined by the HTML spec for
>> <script type="module">, a goal is also to validate that spec.  As it
>> stands it suffers from the issue we are discussing and attempts to
>> simultaneously load multiple top-level modules but does not correctly
>> handle errors in instantiation.
> 
> Can you point me at the parts of the HTML spec. that involved here?
>  ModuleDeclarationInstantiation is a semantics of the ES spec. and it
> isn’t clear to me why the HTML spec. would need to invoke it. From you
> description, I suspect there may be some impedance mismatches between
> the specs.
> 
>>
>> In the browser, this simultaneous loading is important for performance
>> reasons.  Using modules already has the potential to create many more
>> server round trips as dependencies are fetched, so browsers will want to
>> parallelise these fetches as much as possible.  This means parsing
>> modules as soon as they have been fetched to extract their dependencies
>> and starting to fetch their dependencies straight away, all across
>> multiple module trees.
> 
> Shouldn't be a problem.  Just like some  browsers now  eagerly (or
> lazily) parse scripts on a separate thread they should be able to parse
> individual module files as they fetch them and cache parse results (eg,
> ASTs). Even share cached parse results across multiple windows.
>  Fetching and mechanism of how a parsed module or script is internally
> represented is outside of the scope of the ES spec. Eager parsing is
> explicitly allowed for in the spec. For example, see the NOTE
> in https://tc39.github.io/ecma262/#sec-toplevelmoduleevaluationjob. Basically
> an implementation can cache or do anything it wants as long as the fact
> it is doing so is not observable to ES code. The non-observability
> requirement is the reason it says that error reporting most be deferred
> until the explicit parse step in the spec. is reached.
> 
>>
>> I think it's important to outline how this is going to work in the HTML
>> spec itself, especially since the algorithm chosen can have subtle
>> effects on the behaviour.
>>
>> The approach you suggested above would work but I think it would have
>> the disadvantage that while while the transaction for one module tree
>> was in progress it would delay modules from a second module tree being
>> parsed (and their dependencies fetched).  Did I understand that correctly?
> 
> That’s certainly not the intent.  Note
> that https://tc39.github.io/ecma262/#sec-hostresolveimportedmodule is
> required to be idempotent if it completes normally. An implication of
> this is that any complete and valid closed subgraph of a module tree
> should be reusable and sharable among other module trees, even if the
> first module tree to process it fails because other subtree are
> incomplete.  Processing multiple module trees in parallel certainly
> complicates caching and error recovery. But should be possible.  The ES
> spec. really doesn’t care, as long as the implementation specific quirks
> aren’t observable to ES code.
> 
> (BTW, a loader API that includes the ability of ES code to
> programmatically introduce additional modules will also complicate eager
> or parallel processing.  If that is possible, then failure of eager
> processing of some module trees might be a temporary condition, and
> failure means that the processing of the module tree needs to be
> deferred until the API causes something to change.    In other words,
> statically fetched modules that have dependencies upon dynamically
> created modules are tricky )
> 
>>
>> Another similar approach that also uses the committed state might be to
>> parse all modules as they become available and discard any uncommitted
>> dependencies on instantiation error.  Dependencies of a top-level module
>> are marked as committed on successful load.  I think this would be
>> equivalent to some serial execution of TopLevelModuleEvaluationJob for
>> each top-level module.
> 
> All you really need to throw away are newly created Module Environment
> Records as they are the entities that are observable to the ES
> evaluation semantics. But remember to distingish the implementation’s
> internal data structures from the specification’s abstract data
> structures such as Module Records, and Module Environment Records. The
> implementation is free to use any valid memorization it can come up with. 
> 
> Note that the possibility of eager linking/instantiation is why module
> evaluation is a separate step and not included in module instantiation.
>  The job queue for TopLevelEvaluationJobs is there to enable the host
> environment to specify the serial execution order of top level module
> (and script) evaluations.
> 
>>
>> However that still has the disadvantage that too many modules are thrown
>> away on error.  This means that some modules might have to be re-fetched
>> later if they were used by a sequent load, even if they were present
>> when that load started.  My concern is mainly that handling this corner
>> case would complicate the HTML spec.
> 
> Don’t see why refetching would be required (assuming the remote resource
> doesn’t change).  Might have to relink subtrees that had earlier had
> linkage failures.
>>
>> I guess what I'd really like is some support for this simultaneous
>> loading in ES.  Removing the restriction on discarding modules after
>> failure and making ModuleDeclarationInstantiation fail if called again
>> after failure would be one way.  Another would be to report to the
>> caller which modules failed to instantiate.  Either way the problem is
>> that it's hard to handle errors from ModuleDeclarationInstantiation if
>> you don't know which modules failed.
> 
> My sense is that you are seeing requirements/restrictions in the ES
> spec. that aren't there (or aren’t intended if they are).
> 
> We should probably focus on individual issues, one at a time.
> 
> Allen
>