Paging through a collection

Ryan Kelly rfkelly at mozilla.com
Mon Sep 29 18:09:00 PDT 2014


On 30/09/2014 1:55 AM, Richard Newman wrote:
> On  29 Sep 2014, at 8:12 AM, Stefan Arentz <sarentz at mozilla.com> wrote:
> 
>> I don’t understand the logic here. Why doesn’t the client use the limit and offset parameters to grab all history in three requests?
> 
> You’re exercising an old code path that was intended for use by mobile.
> 
> Nobody has stripped this out, because the risk of introducing bugs is higher than the benefit of simplification.

Also note that the way pagination works is different in sync1.5 than it
was in sync1.1, and there's no client code that actually does things the
sync1.5 way.

> Sync doesn’t use limit and offset for several reasons:
> 
> * Limit and offset don’t really make sense on the server. This is not a transactional system; it’s stateless, with an arbitrary amount of time between successive page fetches. The client can’t trust that it’ll get all the records by using paging, so it doesn’t use it at all.
> 
> * Server (and client!) writes could be occurring during this paging behavior. We don’t want to make Sync’s lack of safety worse by deliberately fast-forwarding past new records. You can see in engines.js that we set the timestamp before we start fetching batches.
> 
> * This batching persists across restarts. Paging would not (and wouldn’t make sense to do so, for the aforementioned reasons). This is/was important for XUL mobile.

The next pagination system does a better job of this than in old sync,
through a combination of X-If-Unmodified-Since and opaque offset tokens.
 The API docs include a nice example of how to do it:


http://docs.services.mozilla.com/storage/apis-1.5.html#example-paging-through-a-large-set-of-items

It's not perfect (e.g. there could still be concurrent client-side
writes going on) but it does go some way to addressing the issues
Richard outlined above.


  Ryan


More information about the Sync-dev mailing list