[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH v2 12/17] qcow2: convert QCow2 to use QCryptoBlo
|
From: |
Daniel P. Berrange |
|
Subject: |
Re: [Qemu-devel] [PATCH v2 12/17] qcow2: convert QCow2 to use QCryptoBlock for encryption |
|
Date: |
Tue, 9 Feb 2016 12:32:53 +0000 |
|
User-agent: |
Mutt/1.5.24 (2015-08-30) |
On Mon, Feb 08, 2016 at 11:12:37AM -0700, Eric Blake wrote:
> On 01/20/2016 10:38 AM, Daniel P. Berrange wrote:
> > This converts the qcow2 driver to make use of the QCryptoBlock
> > APIs for encrypting image content. As well as continued support
> > for the legacy QCow2 encryption format, the appealing benefit
> > is that it enables support for the LUKS format inside qcow2.
>
> I know Fam had some interesting ideas on changing qcow2 to be able to
> auto-load a LUKS format driver rather than duplicating code, which may
> completely change this patch, but I'll go ahead and review this patch as-is.
> > (Full Disk Encryption) header extension that specifies
> > the offset of a set of clusters to hold the FDE headers,
> > as well as the length of that region. The LUKS header is
> > thus stored in these extra allocated clusters before the
> > main image payload.
> >
> > With this change it is now required to use the QCryptoSecret
> > object for providing passwords, instead of the current block
> > password APIs / interactive prompting.
> >
> > $QEMU \
> > -object secret,id=sec0,filename=/home/berrange/encrypted.pw \
> > -drive file=/home/berrange/encrypted.qcow2,key-secret=sec0
> >
> > The new LUKS format is set as the new default format when
> > creating encrypted images. ie
> >
> > # qemu-img create --object secret,data=123456,id=sec0 \
> > -f qcow2 -o encryption,key-secret=sec0 \
> > test.qcow2 10G
>
> Did we add '-o encryption' earlier in the series, or is this line a bit
> stale in reference to your --image-opts series?
'-o encryption' is actually the pre-existing flag used to turn
on the existing qcow2 built-in AES encryption. I repurposed it
as a generic flag to indicate desire for encryption, and have
the new 'encryption-format' flag to say whether you want the
legacy AES or modern LUKS format. IOW people using '-o encryption'
today will automatically start getting LUKS format after this
change is applied.
> > diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
> > index f5bc4f2..0512256 100644
> > --- a/block/qcow2-cluster.c
>
> > +++ b/block/qcow2-refcount.c
> > @@ -1847,6 +1847,16 @@ static int calculate_refcounts(BlockDriverState *bs,
> > BdrvCheckResult *res,
> > return ret;
> > }
> >
> > + /* encryption */
> > + if (s->crypto_header.length) {
> > + ret = inc_refcounts(bs, res, refcount_table, nb_clusters,
> > + s->crypto_header.offset,
> > + s->crypto_header.length);
> > + if (ret < 0) {
> > + return ret;
> > + }
> > + }
> > +
>
> Do we ever allow turning off encryption, where we'd need to decrement
> the refcounts as the FDE header is removed?
No, once a fle has encryption it can't be removed. Likewise you have
to choose it at time of creation - you can't add it afterwards.
> > diff --git a/block/qcow2.c b/block/qcow2.c
> > index 2fae692..5249ca2 100644
> > --- a/block/qcow2.c
> > +++ b/block/qcow2.c
> > @@ -34,6 +34,8 @@
> > #include "qapi-event.h"
> > #include "trace.h"
> > #include "qemu/option_int.h"
> > +#include "qapi/opts-visitor.h"
> > +#include "qapi-visit.h"
> >
> > /*
> > Differences with QCOW:
> > @@ -60,6 +62,7 @@ typedef struct {
> > #define QCOW2_EXT_MAGIC_END 0
> > #define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA
> > #define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857
> > +#define QCOW2_EXT_MAGIC_CRYPTO_HEADER 0x4c554b53
>
> Aha: ASCII "LUKS", even though we expect the header to be useful for
> more than just LUKS encryption schemes. Would ASCII "CRYP" be any
> nicer? Or a random number, or the next set of hex digits from pi, or...?
>
> I guess we don't have any real rules or patterns describing how magic
> numbers are supposed to be chosen, so it works for me.
Hah, top marks for reverse engineering my magic constant. I had
completely forgotten that I used the ASCII values of 'LUKS' for
this constant :-)
Since this is not luks specific anymore I'll change this value
to 0x0537be77. Bonus points to anyone who can explain the
significance of that number (hint it is not an ascii representation
of anything - its a special number in its own right :-)
> > +static ssize_t qcow2_crypto_hdr_read_func(QCryptoBlock *block, size_t
> > offset,
> > + uint8_t *buf, size_t buflen,
> > + Error **errp, void *opaque)
> > +{
> > + BlockDriverState *bs = opaque;
> > + BDRVQcow2State *s = bs->opaque;
> > + ssize_t ret;
> > +
> > + if ((offset + buflen) > s->crypto_header.length) {
>
> The inner () are redundant, but I don't mind them.
>
> > + error_setg(errp, "Request for data outside of extension header");
> > + return -1;
>
> Nice that you are trying to prevent reads beyond the block of data
> reserved by the FDE header; but is s->crypto_header.length the
> rounded-up cluster boundary, or just the length of the used portion of
> the LUKS header?...
It is intended to record the allocated space (ie rounded to the cluster
boundary), but on reflection, I think I'll change it to exactly the
size of the LUKS header.
> > + }
> > +
> > + ret = bdrv_pread(bs->file->bs,
> > + s->crypto_header.offset + offset, buf, buflen);
> > + if (ret < 0) {
> > + error_setg_errno(errp, -ret, "Could not read encryption header");
> > + return -1;
> > + }
> > + return ret;
> > +}
> > +
> > +
> > +static ssize_t qcow2_crypto_hdr_init_func(QCryptoBlock *block, size_t
> > headerlen,
> > + Error **errp, void *opaque)
> > +{
> > + BlockDriverState *bs = opaque;
> > + BDRVQcow2State *s = bs->opaque;
> > + int64_t ret;
> > +
> > + s->crypto_header.length = headerlen + (headerlen % s->cluster_size);
>
> ...Huh? Are you trying to round up to a cluster boundary? This doesn't
> do what you want. And even if you had correctly rounded up to cluster
> boundary size, that means that qcow2_crypto_hdr_read_func() can now read
> the tail beyond the size recorded in the crypto header - which could be
> dangerous if it means that tail can overlap the same sector number used
> for the first guest payload sector which uses a sector number of
> payload_offset from the LUKS header for its IV.>
> > +
> > + ret = qcow2_alloc_clusters(bs, s->crypto_header.length);
>
> Does qcow2_alloc_clusters(bs, headerlen) properly round up the
> allocation to a cluster boundary? If so, I think you want
> s->crypto_header.length = headerlen, full stop.
Agreed, I will make it so.
>
> > + if (ret < 0) {
> > + s->crypto_header.length = 0;
>
> And it may be easier to not modify s->crypto_header.length except on
> successful allocation, rather than having to undo it on failure.
>
> > + error_setg_errno(errp, -ret,
> > + "Cannot allocate cluster for LUKS header size
> > %zu",
> > + headerlen);
> > + return -1;
> > + }
> > +
> > + s->crypto_header.offset = ret;
> > + return ret;
> > +}
> > +
> > +
> > +static ssize_t qcow2_crypto_hdr_write_func(QCryptoBlock *block, size_t
> > offset,
> > + const uint8_t *buf, size_t
> > buflen,
> > + Error **errp, void *opaque)
> > +{
> > + BlockDriverState *bs = opaque;
> > + BDRVQcow2State *s = bs->opaque;
> > + ssize_t ret;
> > +
> > + if ((offset + buflen) > s->crypto_header.length) {
> > + error_setg(errp, "Request for data outside of extension header");
>
> Again, I think you want to be very careful and not allow writes beyond
> the initial header length reservation, even if that leaves the tail of
> the cluster-aligned FDE area unwritable.
Yes, this will be ok when the length refers to the luks header only
> > {
> > BDRVQcow2State *s = bs->opaque;
> > @@ -159,6 +232,39 @@ static int qcow2_read_extensions(BlockDriverState *bs,
> > uint64_t start_offset,
> > }
> > break;
> >
> > + case QCOW2_EXT_MAGIC_CRYPTO_HEADER: {
> > + unsigned int cflags = 0;
> > + if (s->crypt_method_header != QCOW_CRYPT_LUKS) {
> > + error_setg(errp, "CRYPTO header extension only "
> > + "expected with LUKS encryption method");
> > + return -EINVAL;
> > + }
> > + if (ext.len != sizeof(Qcow2CryptoHeaderExtension)) {
> > + error_setg(errp, "CRYPTO header extension size %u, "
> > + "but expected size %zu", ext.len,
> > + sizeof(Qcow2CryptoHeaderExtension));
> > + return -EINVAL;
> > + }
> > +
> > + ret = bdrv_pread(bs->file->bs, offset, &s->crypto_header,
> > ext.len);
> > + if (ret < 0) {
> > + error_setg_errno(errp, -ret,
> > + "Unable to read CRYPTO header extension");
> > + return ret;
> > + }
> > + be64_to_cpu(s->crypto_header.offset);
> > + be64_to_cpu(s->crypto_header.length);
>
> Seeing this made me look back at patch 7. It's convenient that the LUKS
> header also uses big-endian storage (otherwise, you'd have to tweak the
> portion of the qcow2 spec that requires big-endian everywhere to special
> case LUKS header content - although that is not the content being
> byte-swapped here).
Yes its fortunate everyone agrees that big-endian is best on disk :-)
> > +
> > + if (flags & BDRV_O_NO_IO) {
> > + cflags |= QCRYPTO_BLOCK_OPEN_NO_IO;
> > + }
> > + s->crypto = qcrypto_block_open(s->crypto_opts,
> > + qcow2_crypto_hdr_read_func,
> > + bs, cflags, errp);
> > + if (!s->crypto) {
> > + return -EINVAL;
> > + }
> > + } break;
> > default:
> > /* unknown magic - save it in case we need to rewrite the
> > header */
>
> Hmm. Based solely on the presence or absence of unknown headers, earlier
> versions of qemu-img that don't recognize the new FDE extension header
> would happily open an image without decrypting it, which may have
> disastrous results if it writes to guest data. But I think we are safe
> in that the only time the FDE extension header is present is if the
> crypt_method is 2, but older versions of qemu should reject crypt_method
> 2 as unknown. Phew - we don't have to burn an incompatible feature bit
> to protect newer images from being corrupted by an older qemu.
Yeah, the crypt_method value saves us, because all older versions are
careful to validate that.
> > +
> > +static QCryptoBlockOpenOptions *
> > +qcow2_crypto_open_opts_init(QCryptoBlockFormat format, QemuOpts *opts,
>
> qemu style doesn't usually split return type from function name. I won't
> request a reformat, but others might be more picky.
The lines get rather long if I dont :-)
> > @@ -754,6 +971,28 @@ static int
> > qcow2_update_options_prepare(BlockDriverState *bs,
> > r->discard_passthrough[QCOW2_DISCARD_OTHER] =
> > qemu_opt_get_bool(opts, QCOW2_OPT_DISCARD_OTHER, false);
> >
> > + switch (s->crypt_method_header) {
> > + case QCOW_CRYPT_NONE:
> > + break;
>
> Are we allowing in-place updates to change the encryption method? It
> may be safer to require that in-place updates can't change encryption,
> for now (that is, you can only set encryption at creation of a new file,
> and then copy data from one file to another to change how it is
> encrypted). And even if we DO want to allow a user to convert an image
> from encrypted to plain, or from plain to encrypted, I would favor it as
> a separate patch, so that we make sure we properly handle the
> creation/removal of the FDE extension, as well as en/decrypting every
> byte of payload independently from simpler read/write of an encrypted image.
I don't see us ever allowing in-place changes to encryption. You'd have
to re-write the entire file contents with obvious disaster if you hit
an error 1/2 way through. Far better to use qemu-img convert instead.
> > static void qcow2_update_options_abort(BlockDriverState *bs,
> > @@ -799,6 +1041,7 @@ static void
> > qcow2_update_options_abort(BlockDriverState *bs,
> > if (r->refcount_block_cache) {
> > qcow2_cache_destroy(bs, r->refcount_block_cache);
> > }
> > + qapi_free_QCryptoBlockOpenOptions(r->crypto_opts);
>
> Again, I don't know that we should allow updating the use of crypto
> during an update options task, or if we do, it should be in a separate
> patch (let's get reading/writing to an encrypted image correct first,
> before we worry about in-place conversion).
Note despite the name the qcow2_update_options* methods are
actually used during initial image creation, so this code is
about updating in-place options.
> > + * Using BDRV_O_NO_IO, since encryption is now setup we don't want to
> > + * have to setup decryption context. We're not doing any I/O on the top
> > + * level BlockDriverState, only lower layers, where BDRV_O_NO_IO does
> > + * not have effect.
> > + */
> > options = qdict_new();
> > qdict_put(options, "driver", qstring_from_str("qcow2"));
> > ret = bdrv_open(&bs, filename, NULL, options,
> > - BDRV_O_RDWR | BDRV_O_CACHE_WB | BDRV_O_NO_BACKING,
> > + BDRV_O_RDWR | BDRV_O_CACHE_WB | BDRV_O_NO_BACKING |
> > + BDRV_O_NO_IO, /* Don't want to activate encryption */
>
> Do we really need the second comment, after the first?
Not any more, no.
> > @@ -3046,9 +3366,9 @@ static int qcow2_amend_options(BlockDriverState *bs,
> > QemuOpts *opts,
> > backing_format = qemu_opt_get(opts, BLOCK_OPT_BACKING_FMT);
> > } else if (!strcmp(desc->name, BLOCK_OPT_ENCRYPT)) {
> > encrypt = qemu_opt_get_bool(opts, BLOCK_OPT_ENCRYPT,
> > - !!s->cipher);
> > + !!s->crypto);
> >
> > - if (encrypt != !!s->cipher) {
> > + if (encrypt != !!s->crypto) {
> > error_report("Changing the encryption flag is not
> > supported");
>
> Oh, so we don't support changing the encryptiong yet (good). But it
> means I was confused about what was happening above with regards to code
> on changing encryption metadata.
Yes, its a misleading method name earlier used for 2 different things :-)
> > @@ -166,6 +168,78 @@ the header extension data. Each entry look like this:
> > terminated if it has full length)
> >
> >
> > +== Full disk encryption header pointer ==
> > +
> > +The full disk encryption header must be present if, and only if, the
> > +'crypt_method' header requires metadata. Currently this is only true
> > +of the 'LUKS' crypt method. The header extension must be absent for
> > +other methods.
> > +
> > +This header provides the offset at which the crypt method can store
> > +its additional data, as well as the length of such data.
> > +
> > + Byte 0 - 7: Offset into the image file at which the encryption
> > + header starts. Must be aligned to a cluster boundary.
> > + Byte 8 - 16: Length of the encryption header. Must be a multiple
> > + of the cluster size.
>
> I would add "in bytes" to both descriptions.
Ok.
> Should we explicitly document that the length occupied by the extension
> header is NOT counted towards the guest-visible length; therefore, the
> guest-visible size is the payload of the encrypted disk?
Sure, can do.
> > +While the header extension allocates the length as a multiple of the
> > +cluster size, the encryption header may not consume the full permitted
> > +allocation.
>
> Should we require that any unused trailing space be all 0?
I guess it might be a nice idea to have it zeroed in case it becomes
important later.
> > +For the LUKS crypt method, the encryption header works as follows.
> > +
> > +The first 592 bytes of the header clusters will contain the LUKS
> > +partition header. This is then followed by the key material data areas.
> > +The size of the key material data areas is determined by the number of
> > +stripes in the key slot and key size. Refer to the LUKS format
> > +specification ('docs/on-disk-format.pdf' in the cryptsetup source
> > +package) for details of the LUKS partition header format.
> > +
> > +In the LUKS partition header, the "payload-offset" field does not refer
> > +to the offset of the QCow2 payload. Instead it simply refers to the
> > +total required length of the LUKS header plus key material regions.
>
> I'm guessing that you are allowing the payload-offset to NOT be a
> multiple of the cluster size. It may be worth documenting that since
> LUKS requires encryption to happen on 512-byte sector boundaries, where
> the sector number feeds the IV used to encrypt that sector, that
> guest-visible sector 0 is treated as LUKS sector payload-offset, even if
> payload-offset is not qcow2-cluster-aligned.
Yeah, there's no restriction on payload-offset at all. Essentially
the payload-offset is completely redundant when used in the context
of qcow2. We could easily have said it should be 0, since qcow2
determines the actual payload offset. I felt it was worth leving
it to record the value LUKS would ordinarily have used, even though
we don't need it.
> > +In the LUKS key slots header, the "key-material-offset" is relative
> > +to the start of the LUKS header clusters in the qcow2 container,
> > +not the start of the qcow2 file.
> > +
> > +Logically the layout looks like
> > +
>
> How does encryption interact with internal snapshots? Is the encryption
> header over all snapshots at once, or are we taking snapshots of the
> current key slot usage? That is, if we later add a command to allow key
> slot manipulation (add a new user password on a vacant key slot, or
> revoke a key slot), what happens if the user takes a snapshot, revokes a
> key slot, then takes a second snapshot, then wants to revert to the
> first snapshot? Will the revoked password still be usable to unlock the
> first snapshot contents, or did revoking it destroy that user's ability
> to access the snapshot?
>
> I'm leaning towards the latter - there is only a single LUKS header for
> the entire qcow2 file; LUKS key management is global regardless of what
> ever other content is stored, and the user passwords that unlock the
> LUKS master key control whether a user can access all or none of the
> rest of the qcow2 data.
NB passwords have no direct relationship to the data encryption key.
The passwords are just used to encrypt the master key. So you can
change the passwords at any time, without loosing ability to
decrypt old data. IOW, we can completely ignore the issue of snapshots.
All snapshot data is using the same master key.
> I suspect that backing files are NOT encrypted by the LUKS header of the
> active file. That is, if we go to read a sector, and it is not present
> in the current qcow2 image, then reading it from the backing file does
> NOT need to decrypt data (unless the backing file itself independently
> used a LUKS header).
Yep, whether any backing file uses encryption is independant of
the decision wrt encryption for this file.
> Conversely, if we have a backing file that is encrypted, do we want to
> place any requirements that the wrapper file can/must use encryption? I
> don't see any technical reasons that would require it, but from a data
> safety standpoint, it seems awkward that a user that provides the LUKS
> password to read the backing file can then write the same data
> unencrypted into their wrapper file on copy-on-write operations.
This feels like a policy decision that belongs in the mgmt app - indeed
that situation already exists today. ie you could have created a qcow2
file that points to a /dev/mapper/BLAH that is in fact backed by a LUKS
volume.
> > +++ b/qapi/block-core.json
> > @@ -1789,6 +1789,8 @@
> > # @cache-clean-interval: #optional clean unused entries in the L2 and
> > refcount
> > # caches. The interval is in seconds. The default
> > value
> > # is 0 and it disables this feature (since 2.5)
> > +# @key-secret: #optional the ID of a QCryptoSecret object
> > providing
> > +# the decryption key (since 2.6)
>
> As in other patches in this series, may be worth mentioning that this is
> mandatory for doing I/O on an encrypted disk, and optional when the disk
> is not encrypted or when only metadata is being probed.
Yes
> > diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out
> > index a2b6703..c9f0bc5 100644
> > --- a/tests/qemu-iotests/049.out
> > +++ b/tests/qemu-iotests/049.out
> > @@ -186,14 +186,11 @@ Formatting 'TEST_DIR/t.qcow2', fmt=qcow2
> > size=67108864 encryption=off cluster_si
> > qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M
> > Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off
> > cluster_size=65536 lazy_refcounts=off refcount_bits=16
> >
> > -qemu-img create -f qcow2 -o encryption=on TEST_DIR/t.qcow2 64M
> > +qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o
> > encryption=on,key-secret=sec0 TEST_DIR/t.qcow2 64M
> > qemu-img: Encrypted images are deprecated
> > Support for them will be removed in a future release.
> > You can use 'qemu-img convert' to convert your image to an unencrypted one.
> > -qemu-img: Encrypted images are deprecated
> > -Support for them will be removed in a future release.
> > -You can use 'qemu-img convert' to convert your image to an unencrypted one.
> > -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on
> > cluster_size=65536 lazy_refcounts=off refcount_bits=16
> > +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on
> > cluster_size=65536 lazy_refcounts=off refcount_bits=16 key-secret=sec0
>
> So now that we support LUKS encryption by default, we no longer need the
> deprecation warning. Do we still forbid the creation of new images with
> non-LUKS encryption? That is, even though the new code will let us read
> old images, I want to make sure we test the error message (or
> deprecation warning) when trying to use the new options to request the
> old format during image creation.
Forbidding creation of legacy encryption is dealt with in a later patch.
> > == Check lazy_refcounts option (only with v3) ==
> >
> > diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
> > index a952330..b0572d4 100644
> > --- a/tests/qemu-iotests/082.out
> > +++ b/tests/qemu-iotests/082.out
> > @@ -53,6 +53,13 @@ cluster_size qcow2 cluster size
> > preallocation Preallocation mode (allowed values: off, metadata,
> > falloc, full)
> > lazy_refcounts Postpone refcount updates
> > refcount_bits Width of a reference count entry in bits
> > +encryption-format Encryption format, 'luks' (default), 'qcow' (deprecated)
> > +key-secret ID of the secret that provides the encryption key
> > +cipher-alg Name of encryption cipher algorithm
> > +cipher-mode Name of encryption cipher mode
> > +ivgen-alg Name of IV generator algorithm
> > +ivgen-hash-alg Name of IV generator hash algorithm
> > +hash-alg Name of encryption hash algorithm
> > nocow Turn off copy-on-write (valid only on btrfs)
>
> Should this help text mention defaults?
Yeah, good idea.
> > +++ b/tests/qemu-iotests/087
> > @@ -184,9 +184,18 @@ echo
> > echo === Encrypted image ===
> > echo
> >
> > -_make_test_img -o encryption=on $size
> > +_make_test_img --object secret,id=sec0,data=123456 -o
> > encryption=on,key-secret=sec0 $size
> > run_qemu -S <<EOF
> > { "execute": "qmp_capabilities" }
> > +{ "execute": "object-add",
> > + "arguments": {
> > + "qom-type": "secret",
> > + "id": "sec0",
> > + "props": {
> > + "data": "123456"
> > + }
> > + }
> > +}
> > { "execute": "blockdev-add",
> > "arguments": {
> > "options": {
> > @@ -195,7 +204,8 @@ run_qemu -S <<EOF
> > "file": {
> > "driver": "file",
> > "filename": "$TEST_IMG"
> > - }
> > + },
> > + "key-secret": "sec0"
> > }
>
> Nice - proof that we can hot-plug a secret. Which means libvirt will
> have to be taught to _always_ prepopulate a master key for new enough
> qemu, even if there is no encrypted disk on the command line, to cater
> for hotplug down the road.
You can actually hotplug the master key too, but for simplicity I
expect that libvirt will just generate a AES master key for all
QEMU versions >= 2.6. We need it for encrypting RBD, iSCSI, Curl
passwords too.
> > +++ b/tests/qemu-iotests/134
> > @@ -44,23 +44,31 @@ _supported_os Linux
> >
>
> > -128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> > +can't open device
> > /home/berrange/src/virt/qemu/tests/qemu-iotests/scratch/t.qcow2: Invalid
> > password, cannot unlock any keyslot
> > +no file open, try 'help open'
>
> Umm, you'll want to fix this test to run on more than just your machine.
Opps :-)
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|