Re: [Nmh-workers] MH + IMAP? (was Large MH directories

[Bakul Shah]

On Oct 14, 2017, at 9:34 AM, Ken Hornstein <address@hidden> wrote:
> 
>> What about storing sequence related info in a separate imap
>> folder in some fake messages?  May be even as a MIME
>> attachments!
>> 
>> 1. Basically map MH msg numbers to Mail-IDs + status in an
>>  "index" email message.  May be one msg per folder.
>> 
>> 2. On any +folder access, this index email would have to be
>>  reconciled with any new or deleted messages.
>> 
>> 3. Initial connection and periodic checking would have to
>>  similarly reconcile with new or deleted folders.
>> 
>> 4. Most MH commands should require no change if messages are
>>  cached locally like now. Basically all local state would
>>  turn into cache and real state stored on the imap server.
>> 
>> 5. An alternative would be to implement most of this in a
>>  separate server & mount the "mailfs" it via fusefs. This is
>>  somewhat analogous of plan9's upas. Such a server would
>>  require no change in MH proper (or may be only for
>>  delivery).
> 
> Some thoughts.
> 
> 1) Anything involving FUSE: no, no, NO, NO!!!!  I have detailed my thoughts
>   on this before, but as a recap: Massively unportable, extremely
>   complicated, simply shifts the hard problems to a more complicated
>   layer.  Let us speak of this no further (unless you want me to elaborate
>   my thoughts on how terrible an idea this is).

:-) Understandable. But see below.

> 2) In terms of implementation details .. I have waffled back and forth
>   about whether or not to store some of the necessary indexing information
>   on the IMAP server or locally.  It occurs to me that storing it locally
>   would be easier to implement, at least for an initial implementation.

Storing this map and metadata locally may be a good start.

> 3) I think the EASIEST thing to do would be to simply access every
>   message on the IMAP server directly, without any local caching.  I
>   realize this isn't ideal, but to me the main issue is developer
>   resources; doing local caching would complicate things tremendously,
>   and that just means more time.  Maybe local caching for offline work
>   could be v2 of the IMAP implementation.

If common MH commands have to connect to an imap server &
login every time, people will notice the delay. May make sense
to have a local imap<->MH daemon. You then connect to it over
a local socket. The daemon makes it easy to add caching.

Actually there is no need for fusefs. The mh-imap daemon can
create the local folders/message files as needed. My assumption
here is that an mh command from a particular user would kick off
the daemon in bg if it is not already running. And the daemon
can exit if there is no use for N minutes.

> 4) I believe a number of operations would actually have improved performance;
>   for example, having pick(1) push it's searches to the IMAP server would
>   be a huge win.  Operations on a larger folder would probably work better.
>   But how things would work overall in terms of performance?  We'd have
>   to see.

Agree.

> 
>> There are a number of benefits of doing this but an MH specific
>> "storage" API would've to be defined first.
> 
> Yeah, that's part of the deal; I've proposed some simple ideas in that
> direction on here before, but they're not well-formed.

I think mh-imap daemon V1 as outlined above would require very
few changes in nmh proper. Basically you have to tell mh-imap
when you perform any operation that changes the message number
mapping (or let it do it for you). But it doesn't care what is
your current folder or current message or which sequences you
have defined.

> 
> I have managed to convince myself that there are no FUNDAMENTAL problems
> with an nmh<->IMAP interface; we just need to roll up our sleeves and write
> the code.  Sadly, lack of time is the issue there.

One problem I see is if multiple mh<->imap clients (however we
implement them) are concurrently accessing the same remote
message store and decide to map message-IDs to local message
numbers differently. It may be possible to abuse the IMAP
protocol to implement nonblocking synchronization!

Thanks for your comments. Looks like this is feasible and,
may be, not too hard. I'll try to find some time to prototype
this. There a number of open source imap clients in Go that
may make a good starting point.