Please help with writing spec for async JSON APIs

[Bruno Jouhier]

It's true that the newline delimited feed is often "good enough". But it
has a number of drawbacks:

   - It is not JSON. This is especially annoying  during development
   because most of the tools understand real JSON, not this format. For
   example during debugging you often work with small feeds (few entries but
   the entries can be long lines); it is very handy to be able to paste a
   whole feed into an editor and reformat it to visualize (or modify) the
   entries.
   - There is a special MIME type and a special file extension for them
   (see
   https://en.wikipedia.org/wiki/Line_Delimited_JSON#MIME_type_and_file_extensions)
   but I've never seen anyone use them (maybe I'm not curious enough). On the
   other hand I've often seen such feeds saved as `.json` files. So the level
   of interoperability is poor.
   - It does not let you cleanly associate header/trailer metadata with
   your feed. With true JSON, you can design your feed as `{ header: { ... },
   body: [...], trailer: {...} }`. With ldjson you can still do it by
   formatting the header as a first line and the trailer (if any) as a trailer
   line but this is brittle (kinda of CSV in JSON, trailer forces you to test
   every entry).

Note: I'm not screaming for an async/incremental parser API in core JS
because I already have libraries to handle arbitrary large JSON feeds. But
if such an API were standardized I'm sure I would get a more
robust/optimized parser.

Bruno

2015-08-03 22:22 GMT+02:00 Domenic Denicola <d at domenic.me>:

> My understanding of most "streaming JSON" use in Node.js is that it
> actually is "newline-delimited JSON." That is, each line is parsed,
> delivered, and processed as a single chunk containing a JSON value, and the
> streaming nature comes from the incremental processing and backpressure as
> applied in between newline-delimited chunks.
>
> Part of the advantage of this is that it's extremely easy to specify and
> implement how it works.
>
> Bruno seems to indicate he's run in to use cases where it'd be useful to
> process a normal JSON object in a streaming fashion. That seems like a
> harder problem, indeed necessitating a SAX-like API.
>
> > -----Original Message-----
> > From: es-discuss [mailto:es-discuss-bounces at mozilla.org] On Behalf Of
> > Brendan Eich
> > Sent: Sunday, August 2, 2015 21:26
> > To: Bruno Jouhier
> > Cc: es-discuss
> > Subject: Re: Please help with writing spec for async JSON APIs
> >
> > Exactly! Incremental and async, i.e., streaming.
> >
> > XML quickly needed such APIs
> > (https://en.wikipedia.org/wiki/Simple_API_for_XML,
> > https://en.wikipedia.org/wiki/StAX). JSON's in the same boat.
> >
> > /be
> >
> > Bruno Jouhier wrote:
> > > A common use case is large JSON feeds: header + lots of entries +
> > > trailer
> > >
> > > When processing such feeds, you should not bring the whole JSON in
> > > memory all at once. Instead you should process the feed incrementally.
> > >
> > > So, IMO, an alternate API should not be just asynchronous, it should
> > > also be incremental.
> > >
> > > FWIW, I have implemented an incremental/evented parser for V8 with a
> > > simple API. This parser is incremental but not async (because V8
> > > imposes that materialization happen in the main JS thread). But, if
> > > the V8 restriction could be lifted, it could be made async with the
> > > same API. See https://github.com/bjouhier/i-json
> > >
> > > i-json's API is a simple low level API. A more sophisticated solution
> > > would be a duplex stream.
> > >
> > > There was also a long discussion on this topic on node's GitHub:
> > > https://github.com/joyent/node/issues/7543
> > >
> > > Bruno
> > > _______________________________________________
> > > es-discuss mailing list
> > > es-discuss at mozilla.org
> > > https://mail.mozilla.org/listinfo/es-discuss
> > _______________________________________________
> > es-discuss mailing list
> > es-discuss at mozilla.org
> > https://mail.mozilla.org/listinfo/es-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/es-discuss/attachments/20150804/d57bc1b2/attachment-0001.html>