Strategy for supporting large message folders

Joshua Cranmer pidgeot18 at gmail.com
Thu Jan 17 19:49:00 UTC 2013


On 1/17/2013 12:59 PM, Kent James wrote:
> On 1/17/2013 7:29 AM, David Bienvenu wrote:
>> On 1/16/2013 2:16 PM, Kent James wrote:
>>> I was looking today at trying to fix the relatively simple bug 
>>> 793865 "nsIMsgParseMailMsgState.envelopePos must be 64bits to 
>>> support large mbox folders", figuring that was one needed step on a 
>>> relatively small path to support >4GB mbox files in local folders. 
>>> But as I looked into it, we have never resolved the issue of a 32 
>>> bit nsMsgKey, which also equals the message offset in local folders. 
>>> It is a far from trivial issue to make progress on that.
>> There's a msg hdr property called something like "message-offset" 
>> that's used instead of the msgKey as the offset into the local 
>> folder, by the berkeley mailbox store. The msgKey is used for 
>> backwards compatibility if there is no message-offset. Are you saying 
>> the message-offset is never used, or sometimes not used?
>>
>> - David
> What I am saying is that the 32 bit nsMsgKey which equals the offset 
> in the case of local mbox folders, is still used in way too many 
> places as a way to access the particular message in a folder.
>
> If I do an mxr search for nsMsgKey in mail folders, I get "Too many 
> hits, displaying the first 1000" I don't understand how the 
> application is supposed to work with message-offset as an alternative 
> to nsMsgKey in all of those places.
>
> I would love for you to tell me that I am incorrect. But just to pick 
> a few examples out of over 1000:
>
> nsIMsgDBView has "selectMsgByKey(in nsMsgKey key)" which is used in 
> mailContextMenus.js
>
> nsMsgFilterService has the array m_searchHits which stores search hits 
> by their nsMsgKey value
>
> I don't understand how those are supposed to work with a big mbox 
> where the offset has exceeded 4GB and the message-offset needs to be 
> used instead.

nsMsgKey is an opaque 32-bit value that refers to, with a folder, a 
unique message. It has no intrinsic value by itself. The berkeley mbox 
store, if requested to retrieve a message, first looks up the 
message-offset property on a header. If it can't find that property 
(it's a mailbox originally created before this functionality was added, 
say), it falls back on the message key value. So the message key isn't 
necessarily the offset within the mbox file; it can just as well be a 
32-bit counter that starts at 0 and increments by 1 every time a new 
message arrives.

-- 
Joshua Cranmer
News submodule owner
DXR coauthor

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/tb-planning/attachments/20130117/d500d5ce/attachment.html>


More information about the tb-planning mailing list