回复: [PATCH] migration/rdma: Use huge page register VM memory

[LIZHAOXIN1 [李照鑫]]

Yes, 'pc.ram' is the specific name for x86. I have read that
memory_region_allocate_system_memory assigns different names
to other architectures.
Thanks for your reminding.

Regards,
lizhaoxin.

-----邮件原件-----
发件人: Daniel P. Berrangé <berrange@redhat.com> 
发送时间: 2021年6月7日 22:18
收件人: LIZHAOXIN1 [李照鑫] <LIZHAOXIN1@kingsoft.com>
抄送: qemu-devel@nongnu.org; quintela@redhat.com; dgilbert@redhat.com; sunhao2 
[孙昊] <sunhao2@kingsoft.com>; DENGLINWEN [邓林文] <DENGLINWEN@kingsoft.com>; 
YANGFENG1 [杨峰] <YANGFENG1@kingsoft.com>
主题: Re: [PATCH] migration/rdma: Use huge page register VM memory

On Mon, Jun 07, 2021 at 01:57:02PM +0000, LIZHAOXIN1 [李照鑫] wrote:
> When using libvirt for RDMA live migration, if the VM memory is too 
> large, it will take a lot of time to deregister the VM at the source 
> side, resulting in a long downtime (VM 64G, deregister vm time is about 
> 400ms).
>     
> Although the VM's memory uses 2M huge pages, the MLNX driver still 
> uses 4K pages for pin memory, as well as for unpin. So we use huge 
> pages to skip the process of pin memory and unpin memory to reduce downtime.
>    
> The test environment:
> kernel: linux-5.12
> MLNX: ConnectX-4 LX
> libvirt command:
> virsh migrate --live --p2p --persistent --copy-storage-inc 
> --listen-address \
> 0.0.0.0 --rdma-pin-all --migrateuri rdma://192.168.0.2 [VM] 
> qemu+tcp://192.168.0.2/system
>     
> Signed-off-by: lizhaoxin <lizhaoxin1@kingsoft.com>
> 
> diff --git a/migration/rdma.c b/migration/rdma.c index 
> 1cdb4561f3..9823449297 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -1123,13 +1123,26 @@ static int qemu_rdma_reg_whole_ram_blocks(RDMAContext 
> *rdma)
>      RDMALocalBlocks *local = &rdma->local_ram_blocks;
>  
>      for (i = 0; i < local->nb_blocks; i++) {
> -        local->block[i].mr =
> -            ibv_reg_mr(rdma->pd,
> -                    local->block[i].local_host_addr,
> -                    local->block[i].length,
> -                    IBV_ACCESS_LOCAL_WRITE |
> -                    IBV_ACCESS_REMOTE_WRITE
> -                    );
> +        if (strcmp(local->block[i].block_name,"pc.ram") == 0) {

'pc.ram' is an x86 architecture specific name, so this will still leave a 
problem on other architectures I assume.

> +            local->block[i].mr =
> +                ibv_reg_mr(rdma->pd,
> +                        local->block[i].local_host_addr,
> +                        local->block[i].length,
> +                        IBV_ACCESS_LOCAL_WRITE |
> +                        IBV_ACCESS_REMOTE_WRITE |
> +                        IBV_ACCESS_ON_DEMAND |
> +                        IBV_ACCESS_HUGETLB
> +                        );
> +        } else {
> +            local->block[i].mr =
> +                ibv_reg_mr(rdma->pd,
> +                        local->block[i].local_host_addr,
> +                        local->block[i].length,
> +                        IBV_ACCESS_LOCAL_WRITE |
> +                        IBV_ACCESS_REMOTE_WRITE
> +                        );
> +        }
> +
>          if (!local->block[i].mr) {
>              perror("Failed to register local dest ram block!\n");
>              break;

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|