Strategy for supporting large message folders

Kent James kent at caspia.com
Thu Jan 17 23:03:28 UTC 2013


On 1/17/2013 11:49 AM, Joshua Cranmer wrote:
> On 1/17/2013 12:59 PM, Kent James wrote:
>> On 1/17/2013 7:29 AM, David Bienvenu wrote:
>>> On 1/16/2013 2:16 PM, Kent James wrote:
>>>> I was looking today at trying to fix the relatively simple bug 
>>>> 793865 "nsIMsgParseMailMsgState.envelopePos must be 64bits to 
>>>> support large mbox folders", figuring that was one needed step on a 
>>>> relatively small path to support >4GB mbox files in local folders. 
>>>> But as I looked into it, we have never resolved the issue of a 32 
>>>> bit nsMsgKey, which also equals the message offset in local 
>>>> folders. It is a far from trivial issue to make progress on that.
>>> There's a msg hdr property called something like "message-offset" 
>>> that's used instead of the msgKey as the offset into the local 
>>> folder, by the berkeley mailbox store. The msgKey is used for 
>>> backwards compatibility if there is no message-offset. Are you 
>>> saying the message-offset is never used, or sometimes not used?
>>>
>>> - David
>> What I am saying is that the 32 bit nsMsgKey which equals the offset 
>> in the case of local mbox folders, is still used in way too many 
>> places as a way to access the particular message in a folder.
>>
>> If I do an mxr search for nsMsgKey in mail folders, I get "Too many 
>> hits, displaying the first 1000" I don't understand how the 
>> application is supposed to work with message-offset as an alternative 
>> to nsMsgKey in all of those places.
>>
>> I would love for you to tell me that I am incorrect. But just to pick 
>> a few examples out of over 1000:
>>
>> nsIMsgDBView has "selectMsgByKey(in nsMsgKey key)" which is used in 
>> mailContextMenus.js
>>
>> nsMsgFilterService has the array m_searchHits which stores search 
>> hits by their nsMsgKey value
>>
>> I don't understand how those are supposed to work with a big mbox 
>> where the offset has exceeded 4GB and the message-offset needs to be 
>> used instead.
>
> nsMsgKey is an opaque 32-bit value that refers to, with a folder, a 
> unique message. It has no intrinsic value by itself.
and in my own case for Exchange Web Services, I simply increment the key 
by one for each new message.

> The berkeley mbox store, if requested to retrieve a message, first 
> looks up the message-offset property on a header. If it can't find 
> that property (it's a mailbox originally created before this 
> functionality was added, say), it falls back on the message key value. 
> So the message key isn't necessarily the offset within the mbox file; 
> it can just as well be a 32-bit counter that starts at 0 and 
> increments by 1 every time a new message arrives.
I could be, but it isn't. Existing mbox databases have the key equal to 
the offset, right? How do we change that? What happens when the offset 
gets > 4GB and someone is trying to access a particular message in the 
database using the 32 bit key?

I'm not say it is impossible, I'm just saying it is a big, 
regression-prone job that I don't think is worth taking on right now 
(compared to getting maildir working correctly).

:rkent

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/tb-planning/attachments/20130117/3515804d/attachment.html>


More information about the tb-planning mailing list