Currently, the only way to use a self-signed certificate for an IRC server (or bouncer) is to add the certificate to the root certificate list or to disable certificate verification all together.
Adding it to the root CA list means that other certificate signed with the matching private key will also be accepted, which is not desirable since it could be used to forge certificates for existing servers when compromised. Additionally, there is only one global setting for which root CA list to use, and it can't be set per server. That means that you'd have to either add the certificate to your distribution's CA list or keep a copy in in sync somewhere with the added certificate. Adding it to your distribution's CA list would even compromise https and other applications if the private key was ever stolen, so this would be a very bad idea™. Keeping a copy in sync is just a hassle, although it can no doubt be automated.
Disabling certificate verification entirely means you are vulnerable to man-in-the-middle attacks again, which means the whole purpose of SSL/TLS is kind of defeated. Sure, the traffic is encrypted, but with enough effort it can still be eavesdropped on.
A much better option, in my opinion, is to allow the user to specify exactly which certificate is allowed for a specific server. That way you can use a self-signed certificate without fear of compromising traffic to other server and without being susceptible to man-in-the-middle attacks. To keep things easy (for the implementation and for the user) I think that a sha1 fingerprint of the certificate is enough to identify the certificate uniquely and safely.
I added an option irc.server.*.ssl_fingerprint . When set and not an empty string, the only certificate accepted for the server is the one with that fingerprint. It should be the SHA1 hash of the certificate without separators between the bytes, exactly in the format as shown when connecting to the server. Otherwise valid certificates that have been signed by a trusted CA will not be accepted if this option is non-empty, unless of course the fingerprint matches.
I attached the patch. I hope I followed the coding style. Any comments or remarks are welcome.
0001-irc-add-option-to-accept-server-certificate-with-fin.patch