Can JS Algorithm object finally solve html5 codecs gridlock and move us forward?
jussi.kalliokoski at gmail.com
Sat Jan 19 02:06:39 PST 2013
On Fri, Jan 18, 2013 at 11:51 PM, neuralll <neuralll at gmail.com> wrote:
> Hi guys
> I am playing with web audio api and mozilla audio api but I got frustrated about so much html5 potential wasted due to infinite codecs deadlock support in browsers holding progress back for so long.
> This even resulted to people writing codecs purely in JS.
> Unfortunately performance is obviously slow and limiting to browsers allowing raw data from js
> And slow mainly because of huffman and imdct.
> But looking at block diagrams of pretty much any widespread codec like theora ogg mpg aac mp4
> it striked me.
> Since they pretty much share the same building blocs. and which themself are not that much patent encumbered.
> So why not just introduce new Algorithm object in JS
> and just register huffman idct imdct filterbank in it. for browsers its easy task they just make visible already heavily hw optimized pretty much 4 or so functions that are already in them
I've been trying to address the performance bottlenecks for signal
processing on the web platform a lot lately. Imho, you're on the right
track, approaching this from the angle of adding basic building blocks
rather than complete solutions.
One significant addition for ES6 is Number#clz(), which is currently a
rather significant bottleneck in our codecs.
Aside from that, at Audio WG  we're working on a DSP API  that
in best case scenarios will give you very close to (or sometimes even
better than) native performance for vector processing. Currently the
DSP API includes the DSP interface (add, sub, mul, div, ramp, etc),
the FFT interface and a Filter interface that lets you specify
arbitrary coefficients, optimizing the algorithms under the hood based
on whether you have a biquad filter or long convolution or whatever.
It might be worth adding DCT/MDCT to the DSP API, but actually I
wonder if at least DCT can just be made a special case of FFT without
being very wasteful. This needs some thinking. Anyway, would be great
to get some implementer interest for the DSP API. Current
implementations include a JS polyfill, my node C++ module
implementation ('dsp' on npm) and a partial SpiderMonkey prototype
implementation by Jens Nockert.
Huffman isn't directly related to signals so that probably doesn't
belong in the DSP API. I'm not convinced it really makes sense to have
a native implementation for it anyway... However, wasn't there some
Archive API in the works or something that might have something
David mentioned mentioned parallelization, but at least for audio that
doesn't make much sense, at least unless you're trying to decode all
of a few hours long track into memory as fast as possible, which isn't
a likely scenario. That said, we're moving aurora.js to do the
decoding off the main thread to be less vulnerable to audio interrupts
and blocking the main thread as little as possible, but so far it has
increased our overall CPU usage (as anticipated), because with the
current Audio APIs, we have to transfer the decoded audio to the main
thread for playback. Transferables help significantly, but that
problem has to be solved at the Audio API level.
More information about the es-discuss