Re: [PATCH v4 4/6] iotests: 194: test also migration of dirty bitmap

[Thomas Huth]

On 04/06/2020 09.51, Vladimir Sementsov-Ogievskiy wrote:
> 04.06.2020 10:21, Thomas Huth wrote:
>> On 03/06/2020 10.06, Vladimir Sementsov-Ogievskiy wrote:
>>> 03.06.2020 10:52, Thomas Huth wrote:
>>>> On 22/05/2020 00.06, Vladimir Sementsov-Ogievskiy wrote:
>>>>> Test that dirty bitmap migration works when we deal with mirror.
>>>>>
>>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>>> Reviewed-by: Eric Blake <eblake@redhat.com>
>>>>> ---
>>>>>    tests/qemu-iotests/194     | 14 ++++++++++----
>>>>>    tests/qemu-iotests/194.out |  6 ++++++
>>>>>    2 files changed, 16 insertions(+), 4 deletions(-)
>>>>
>>>>    Hi!
>>>>
>>>> This test broke the iotest in the gitlab CI:
>>>>
>>>>    https://gitlab.com/huth/qemu/-/jobs/578520599#L3780
>>>>
>>>> it works again when I revert this commit.
>>>>
>>>> Could the test be reworked so that it works in CI pipelines, too?
>>>> Otherwise, I think it's best if we disable it in the .gitlab-ci.yml
>>>> file...
>> [...]
>>> - Qemu aborted. Not good. Definitely is better to fix it than just
>>> exclude the test.. I can't reproduce. Could you provide backtrace from
>>> coredump?
>>
>> It aborted in block/dirty-bitmap.c, line 295, that's the
>> "assert(!bdrv_dirty_bitmap_busy(bitmap));" if I got it right.
>>
>> Full backtrace here:
>>
>>   https://gitlab.com/huth/qemu/-/jobs/580553686#L3638
>>
> 
> Aha, missed it, thanks.

No, you didn't miss it. That was a new run where I added some more
debugging magic to the gitlab-ci.yml file:

build-tcg-disabled:
 [...]
 - make -j"$JOBS"
 - sysctl -w kernel.core_pattern=/tmp/core
 - ulimit -c
 - cd tests/qemu-iotests/
 - ./check -raw 194 || find /tmp -name "*core*"
 - gdb -ex bt ../../x86_64-softmmu/qemu-system-x86_64 /tmp/core*

> Hm. in 194 iotest we wait for migration finish on source, but not on
> target. I can assume, that in your setup target shutdown occurs earlier
> than migration finish, so we have busy bitmap on shutdown.

Sounds plausible. Especially if you consider that these gitlab CI
containers only run with one virtual CPU, so there is likely more delay
here.

> It's known bug (at least for me :), patch is in list:
> [PATCH v2 00/22] Fix error handling during bitmap postcopy
>   [..]
>   [PATCH v2 10/22] migration/block-dirty-bitmap: cancel migration on
> shutdown 
> (https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg04288.html)
>      If target is turned of prior to postcopy finished, target crashes
>      because busy bitmaps are found at shutdown.
>      Canceling incoming migration helps, as it removes all unfinished (and
>      therefore busy) bitmaps.
>   [..]
> 
> 
> Still, of course iotest 194 should be fixed too, to wait for migration
> finish on target too, I'll send a patch.

Thanks, I'll give it a try!

 Thomas