rfc2104 change on 2008-02-01

[Simon Josefsson]

Hi.

I have tracked down a problem sending authenticated e-mail that resulted
in randomly failed CRAM-MD5 authentication (it sometimes work, usually
after 2-3 retries).  The cause is:

2008-02-01  Kenichi Handa  <address@hidden>

        * rfc2104.el (rfc2104-hexstring-to-byte-list): Renamed from
        rfc2104-hexstring-to-bitstring and changed to return a byte list.
        (rfc2104-hash): Convert the result of concat to unibyte string.

The significant patch is:

-    (funcall hash (concat k_opad (rfc2104-hexstring-to-bitstring
-                                 ;; perform inner hash
-                                 (funcall hash (concat k_ipad text)))))))
+    (funcall hash
+            (encode-coding-string
+             (concat k_opad (rfc2104-hexstring-to-byte-list
+                             ;; perform inner hash
+                             (funcall hash (concat k_ipad text))))
+             'iso-latin-1))))

The data to be hashed contains binary data, printed it just looks like
garbage.  Presumably, the reason the operation fails is because it can't
be encoded into latin-1 which as far as I recall have some reserved
parts.  This would be consistent with only happening sometimes, since it
may happen that a string doesn't contain any reserved characters.

I have experimented with fixes, and changed the 'iso-latin-1 to 'binary
and that appears to work.

There is nothing iso-latin-1 specific about CRAM-MD5 so if that is the
right thing, I think there should be a comment to explain why.

Handa, could you explain the motivation for this change?

To resolve it, we can chose:

1) Revert the patch.

2) Change 'iso-latin-1 to 'binary.

There may be other options too.

I propose to change it to 'binary if I don't hear anything.

If there were mailing list discussions about this, I'd appreciate a
pointer to that.  I don't think I was cc'ed, and I couldn't find any
discussion now that I search for it.

Thoughts?

Thanks,
Simon