Re: [VM] searching in mime encoded email

[Uday S Reddy]

Ulrich Mueller writes:

> I think that Julian is right here. Folders don't have any specific
> character encoding, they are simply a stream of bytes. (In terms of
> coding systems, it's "raw-text".) Character sets come into play on the
> level of individual messages, and they're specified by the message's
> (or part's in case of multipart messages) MIME headers.

If a folder is simply a stream of bytes then neither M-s nor V-C-text would
make sense.  For searching, the question is how those bytes are interpreted
in Emacs.  Since 7-bit ASCII is common to all encodings (including Emacs's
own internal encoding), searching happens to work for the ASCII parts.  It
doesn't work for the non-ASCII parts.  (So, when I say folders are in
"ASCII", and Julian says they are in "binary", we are really saying the same
thing.  I am focusing on the 7 bits that can be searched.  He is focusing on
the fact that the 8th bit doesn't get munged by Emacs or VM.)

The two questions that were posed in this thread were:

1. how can we save MIME-decoded messages?

2. how can we search in all the text parts of messages?

For the first question, there is no reason at all why VM can't save
MIME-decoded text in files.  If a message arrives using an 8-bit character
set or a multi-byte character set, encoded in base64 or quoted-printable,
it should be possible for VM to produce an equivalent message replacing the
MIME-encoded parts with text/plain parts, so that it can be saved
somewhere.

Assuming we have that working, I started wondering where such decoded
messages can be saved: in individual files only, or other VM folders?  I
think what Julian and you are saying is that it is perfectly fine to save
them in VM folders.  I am coming around to agreeing with that.

However, the second question is now important.  Searching in MIME-encoded
text parts is not feasible with our present tool set, but one would hope
that it should become possible at least in the new folders where
MIME-decoded text has been saved.  But it is still not possible!  It is not
possible for the same reason that you have been crying about.  The folders
are streams of bytes.  So, it is still only the ASCII parts that can be
searched.  Bummer!

So, my idea is to allow VM folders that are *text* files.  Internally in
Emacs, they will be in the Emacs internal coding.  When they are saved to
disk, they would be saved in an encoding that Emacs chooses or the user
chooses.  Full text search should be possible for such folders, including
grep.  Otherwise there would be no point in having such folders.

So, if you guys can put your mind to the question of whether such folders
will work, we can make some progress.

> There's another aspect why general recoding of saved messages might
> not be a good idea: A message can be PGP signed, and any change of
> encoding will destroy the message's integrity and therefore render the
> signature invalid.

I would expect that when the user saves decoded messages, they would strip
the PGP signatures and decrypt any encrypted messages.  Decryption may not
be always advisable, but it depends on the user what he/she wants to do.  

> Before reinventing the wheel, maybe it would be worthwhile to look at
> dedicated search tools like mairix. There's also an Emacs interface
> for it, see <http://randomsample.de/mairix-el-doc/>.

Indeed, can people that use mairix tell us what is feasible with it?  

Cheers,
Uday