Re: [PATCH] migration/rdma: prevent from double free the same mr

[address@hidden]

On 09/07/2021 03:11, Dr. David Alan Gilbert wrote:
> * Li Zhijian (lizhijian@cn.fujitsu.com) wrote:
>> backtrace:
>> '0x00007ffff5f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
>> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
>> 478             void *addr              = mr->addr;
> ANy idea why it deletes the same mr twice?

It's easy to reproduce it if we specify a nvdimm backing to a fsdax 
memory-backend-file which cannot support registering mr like:

[root@iaas-rpma ~]# mount | grep pmem0
/dev/pmem0 on /mnt/pmem0 type ext4 (rw,relatime,seclabel,dax=always)

[root@iaas-rpma ~]# ndctl list -n namespace0.0
[
   {
     "dev":"namespace0.0",
     "mode":"fsdax",
     "map":"mem",
     "size":536870912,
     "sector_size":512,
     "blockdev":"pmem0"
   }
]


`-object 
memory-backend-file,id=mem1,share=on,mem-path=/mnt/pmem0/nv-128m.img,size=128m,pmem=on,align=2m
 -device nvdimm,memdev=mem1,id=nv1`

and then enable rdma-pin-all.

(qemu) migrate_set_capability rdma-pin-all on
(qemu)

Now qemu has at least 2 ram block, pc.ram and mem1. the latter will be failed 
to register mr:
`Failed to register local dest ram block! : Invalid argument   `

in this case, the mr of pc.ram will be deleted twice.

Thanks
Li
>
> Dave
>
>> (gdb) bt
>>   #0  0x00007ffff5f44ec2 in __ibv_dereg_mr_1_1 (mr=0x7fff1007d390) at 
>> /home/lizhijian/rdma-core/libibverbs/verbs.c:478
>>   #1  0x0000555555891fcc in rdma_delete_block (block=<optimized out>, 
>> rdma=0x7fff38176010) at ../migration/rdma.c:691
>>   #2  qemu_rdma_cleanup (rdma=0x7fff38176010) at ../migration/rdma.c:2365
>>   #3  0x00005555558925b0 in qio_channel_rdma_close_rcu (rcu=0x555556b8b6c0) 
>> at ../migration/rdma.c:3073
>>   #4  0x0000555555d652a3 in call_rcu_thread (opaque=opaque@entry=0x0) at 
>> ../util/rcu.c:281
>>   #5  0x0000555555d5edf9 in qemu_thread_start (args=0x7fffe88bb4d0) at 
>> ../util/qemu-thread-posix.c:541
>>   #6  0x00007ffff54c73f9 in start_thread () at /lib64/libpthread.so.0
>>   #7  0x00007ffff53f3b03 in clone () at /lib64/libc.so.6 '
>>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> ---
>>   migration/rdma.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/migration/rdma.c b/migration/rdma.c
>> index b6cc4bef4a8..0f22b8227c0 100644
>> --- a/migration/rdma.c
>> +++ b/migration/rdma.c
>> @@ -1143,6 +1143,7 @@ static int qemu_rdma_reg_whole_ram_blocks(RDMAContext 
>> *rdma)
>>   
>>       for (i--; i >= 0; i--) {
>>           ibv_dereg_mr(local->block[i].mr);
>> +        local->block[i].mr = NULL;
>>           rdma->total_registrations--;
>>       }
>>   
>> -- 
>> 2.30.2
>>
>>
>>