Re: [PATCH] rbd: Use RBD fast-diff for querying actual allocation

[李义]

On 6/9/20, Jason Dillaman <jdillama@redhat.com> wrote:
> On Tue, Jun 9, 2020 at 3:31 AM Yi Li <yili@winhong.com> wrote:
>>
>> Since Ceph version Infernalis (9.2.0) the new fast-diff mechanism
>> of RBD allows for querying actual rbd image usage.
>>
>> Prior to this version there was no easy and fast way to query how
>> much allocation a RBD image had inside a Ceph cluster.
>>
>> To use the fast-diff feature it needs to be enabled per RBD image
>> and is only supported by Ceph cluster running version Infernalis
>> (9.2.0) or newer.
>>
>> Without the fast-diff feature enabled qemu-img will report an allocation
>> identical to the image capacity.
>>
>> 'qemu-img info rbd:cepharm/liyi-rbd' might output for example:
>>
>>   image: json:{"driver": "raw", "file": {"pool": "cepharm",
>>   "image": "liyi-rbd", "driver": "rbd"}}
>>   file format: raw
>>   virtual size: 20 GiB (21474836480 bytes)
>>   disk size: 0 B
>>   cluster_size: 4194304
>>
>> Newly created rbds will have the fast-diff feature enabled.
>>
>> Signed-off-by: Yi Li <yili@winhong.com>
>> ---
>>  block/rbd.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 60 insertions(+)
>>
>> diff --git a/block/rbd.c b/block/rbd.c
>> index 617553b022..f231653f7b 100644
>> --- a/block/rbd.c
>> +++ b/block/rbd.c
>> @@ -1107,6 +1107,65 @@ static int64_t qemu_rbd_getlength(BlockDriverState
>> *bs)
>>      return info.size;
>>  }
>>
>> +#if LIBRBD_VERSION_CODE > 265
>> +static int disk_usage_callback(uint64_t offset, size_t len, int exists,
>> +                               void *arg)
>> +{
>> +  uint64_t *used_size = (uint64_t *)(arg);
>> +  if (exists) {
>> +    (*used_size) += len;
>> +  }
>> +  return 0;
>> +}
>> +#endif
>> +
>> +static int64_t qemu_rbd_allocated_file_size(BlockDriverState *bs)
>> +{
>> +    BDRVRBDState *s = bs->opaque;
>> +    rbd_image_info_t info;
>> +    int r;
>> +    uint64_t used_size = 0;
>> +    uint64_t features = 0;
>> +
>> +    r = rbd_stat(s->image, &info, sizeof(info));
>> +    if (r < 0) {
>> +        return r;
>> +    }
>> +
>> +    r = rbd_get_features(s->image, &features);
>> +    if (r < 0) {
>> +        return r;
>> +    }
>
> You should probably test the flags to ensure that the
> RBD_FLAG_FAST_DIFF_INVALID flag is not set [1]. It's potentially very
> slow and expensive to calculate the disk usage w/o a fast-diff (on
> large images) since it requires iterating over every possible 4MiB
> backing data object (by default) to query its actual usage.
>

Thanks for your review and remind me of this issue.

>> +   /*
>> +    * rbd_diff_iterate2() is available in versions above Ceph 0.94
>> (Hammer)
>> +    * It uses a object map inside Ceph which is faster than
>> rbd_diff_iterate()
>> +    * which iterates all objects.
>> +    * LIBRBD_VERSION_CODE for Ceph 0.94 is 265. In 266 and upwards
>> diff_iterate2
>> +    * is available
>> +    */
>> +#if LIBRBD_VERSION_CODE > 265
>> +    if (features & RBD_FEATURE_FAST_DIFF) {
>> +
>> +        /*
>> +         * RBD image fast-diff feature enabled
>> +         * Querying for actual allocation.
>> +         */
>> +        r = rbd_diff_iterate2(s->image, NULL, 0, info.size, 0, 1,
>> +                              &disk_usage_callback,
>> +                              &used_size);
>> +        if (r < 0) {
>> +            return r;
>> +        }
>> +    } else {
>> +        used_size = info.size;
>> +    }
>> +#else
>> +    used_size = info.size;
>> +#endif
>> +    return used_size;
>> +}
>> +
>>  static int coroutine_fn qemu_rbd_co_truncate(BlockDriverState *bs,
>>                                               int64_t offset,
>>                                               bool exact,
>> @@ -1316,6 +1375,7 @@ static BlockDriver bdrv_rbd = {
>>      .bdrv_get_info          = qemu_rbd_getinfo,
>>      .create_opts            = &qemu_rbd_create_opts,
>>      .bdrv_getlength         = qemu_rbd_getlength,
>> +    .bdrv_get_allocated_file_size = qemu_rbd_allocated_file_size,
>>      .bdrv_co_truncate       = qemu_rbd_co_truncate,
>>      .protocol_name          = "rbd",
>>
>> --
>> 2.25.3
>>
>>
>>
>>
>
> [1]
> https://github.com/libvirt/libvirt/commit/21deeaf02fdf216b08210fc899579736973ca81d#diff-107c5451015e5980c90048ff615becb8
>
> --
> Jason
>
>