Re: IMAP server (was: Re: [Savannah-hackers] gnu-socketserver)

[Nic Ferrier]

>>> "Alain Magloire" <address@hidden> 26-Feb-01 4:40:49 PM >>>

>Somebody asked me if it was possible to write a 
>full JavaMail using mailutils as the back end.  
>The answer was yes.  His ideas were to cut 
>development time by implementing most of the 
>functions with JNI wrappers calling mailutils.
>Although feasible it is not a good idea, JavaMail 
>provides a very nice framework which is _pure_ 
>JAVA(i.e portable).

Javamail provides a horrible framework... I can't stand it
personally.

However JNI would be terribly slow and not very nice. Sun's Javamail
is slow enough without getting JNI involved.

The oje project (at sourceforge but soon part of GNU I hope) is
developing a free (LGPLed) version of Javamail. That's where my IMAP
provider fits in.


>A full IMAP server/client, will be hard on the skin, 
>IMAP4 protocol is complex compare, to say, POP3.
>I'm sure you are quite aware of the rfc2060(IMAP4), 
>the coming IMAP5 and all the protocol extensions 
>(idle, threading, etc ...)

I wasn't aware of IMAP5... is it much different? When's it due? If
not for a while then that's why I haven't heard of it - I don't let
things come on my radar till they've matured a bit  /8->

I've been hacking various IMAP4 tools (mostly in Java) for a couple
of years now. I know the protocol pretty well. 

I think clients are harder than servers, the server just has to
package up the data in it's store into s-expressions and keep a
running overview of thinks like incomming messages. It's quite a nice
protocol from a server point of view. 

Where it gets hard is in parsing all those differents kinds of data.
Why couldn't they just stick to s-expressions for everything? Having
said that clients aren't so difficult in Java because the Kawa system
(a GNU tool) provides an s-expression reader which works fine for IMAP
parsing.


>: - abstracted backend,
>Good, I'm always advocating for decoupling, abstraction 
>layer and define interfaces.

So am I. Nobody listens though  /8->


>: use files as messages or files as folders
>Not sure I follow here, if you have a Unix mailbox 
>format, you are going to explode the messages into files ?

Don't think aboput the traditional mailbox format with this server.
It's a mailstore with only ONE interface: IMAP. All work is carried
out through the protocol interface.

In one implementation of the backend the store would be one directory
per user, each user having under that directory one directory for each
folder and one file for each message. Things like flag options would
be stored in special files not available to the user through the IMAP
interface.

In another implementation the user's mailbox might be trad unix
mailboxes, or MMDF files or some other wierd and wonderfull file
format but it is always hidden completly from the user by IMAP. The
user won't have rights to the filesystem directories which make up the
mailstore.

Personally I'm only interested in the one-file==one-message system
because that will be real fast. It has a downside that it's wastefull
on inodes which is why I'm abstracting the interface to the backend;
people who want to store their mail on an NFS server can write an
inode econmoic backend. Personally I think huge mail stores are a bad
idea... multiple servers work better.

BTW a side effect of my "you only talk to this through IMAP" is that
it eliminates all locking problems. Since there is only one process
accessing the IMAP store we don't have to worry about locking at all.


>: - all delivery through the server, using an EXIM 
> transport agent that speaks IMAP

>Again, not sure I follow, IMAP is primary for 
>dowloading/browsing emails from multiple folders,  
>mail delivery usually go to SMTP.

Yes. I understand that. But there's no reason that you can't have
delivery via IMAP. That's what append is for. The delivery agent just
has to have the rights to do the delivery.

Here's how a delivery agent works:

- exim recieves mail for nferrier
- exim works out that nferrier's mail goes to imap server "nicimap"
- exim calls delivery agent
- delivery agent opens IMAP connection to nicimap
- del-agent logs into IMAP as super user
- del-agent selects folder "/nferrier"
- del-agent appends incomming mail to inbox

At the last stage the user might want some scripts to execute (for
filtering) so we'll pass the incomming mail through some filters
written in Scheme.

Geddit?


>Allright!  Glad to see you've applied some good 
>patterns to your code, do you have an URL handy?

Not yet... I've got about a million things on my plate right now. My
website is massively out of date and it's about 50 on my list of
priorities.

You can get socketserver from subversions if you're interested. The
module name is "socket-server".


>So far, I've never see the behaviour of a server written 
>in pure Java put under heavy load i.e if the webservers 
>of google.com or yahoo and the databases were written 
>in pure java, what would be the performance?

Try www.talk21.com. We've got 3 million subscribers, a *lot* of hits
and a complex app. It's all pure Java (except the IMAP and LDAP
servers).


>I've written some industrial programs with Java in 
>another life but the emphasize was on AWT and nice 
>little windows.  The heavy stuff was still in C/Ada 
>and fortran.

Java on the server side is perfectly acceptable. It's robust and it's
fast. I think you'll be surprised.


>The problem beside the performance concern is 
>security.  Servers usually fork() for security concerns, 
>many daemons have to setuid()/setgid() to do some 
>priviledge operations.  Threads  do not isolate enough
>since it will affect all the other threads with nasty race 
>conditions.

Yes. That's true. Java can't help you there. I'll have to run my
server as root. 

But I'm not thinking about my server in the way that you think about
normal Unix services. I will dedicate a whole server to it (and
probably several servers in the future). 

I think the way I'm doing things will be more attractive to mail
providers serving a *lot* of customers. We'll use it on talk21 for
example where IMAP speed is our biggest problem.


>: Having said all that the reason I'm using Java is 
>:because that's what I use for nearlly everything now
>:To me it's just a good RAD tool.

>Yes, do not get me wrong, although I sound very critical, 
>this is by no mean bashing Java.

You don't sound critical at all. These are valid concerns.


>Interesting, did not BeOS had there mails tigth up with 
>the filesystem ?

No idea. Never used it.


>Thank you for sharing them.  Any URL handy ?
>The project sound very interesting/promessing.

Part of the reason that there's no URL handy is because I've not
pesuaded anyone else to help. I'm trying to get the owner of talk21 to
fund the development (as I could do it myself in about a month).


Nic