Re: [Qemu-block] [Qemu-devel] [PATCH 1/3] iotests: add test 255 to check bitmap life after snapshot + commit

[Vladimir Sementsov-Ogievskiy]

25.05.2019 2:15, John Snow wrote:
> 
> 
> On 5/23/19 11:47 AM, Vladimir Sementsov-Ogievskiy wrote:
>> Two testcases with persistent bitmaps are not added here, as there are
>> bugs to be fixed soon.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
>> ---
>>   python/qemu/__init__.py    |  4 +-
>>   tests/qemu-iotests/255     | 81 ++++++++++++++++++++++++++++++++++++++
>>   tests/qemu-iotests/255.out | 17 ++++++++
>>   tests/qemu-iotests/group   |  1 +
>>   4 files changed, 102 insertions(+), 1 deletion(-)
>>   create mode 100755 tests/qemu-iotests/255
>>   create mode 100644 tests/qemu-iotests/255.out
>>
>> diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
>> index 81d9657ec0..4c4317a55e 100644
>> --- a/python/qemu/__init__.py
>> +++ b/python/qemu/__init__.py
>> @@ -402,7 +402,7 @@ class QEMUMachine(object):
>>           self._qmp.clear_events()
>>           return events
>>   
>> -    def event_wait(self, name, timeout=60.0, match=None):
>> +    def event_wait(self, name, timeout=60.0, match=None, fail_on=None):
>>           """
>>           Wait for specified timeout on named event in QMP; optionally filter
>>           results by match.
>> @@ -430,6 +430,7 @@ class QEMUMachine(object):
>>   
>>           # Search cached events
>>           for event in self._events:
>> +            assert event['event'] != fail_on
>>               if (event['event'] == name) and event_match(event, match):
>>                   self._events.remove(event)
>>                   return event
>> @@ -437,6 +438,7 @@ class QEMUMachine(object):
>>           # Poll for new events
>>           while True:
>>               event = self._qmp.pull_event(wait=timeout)
>> +            assert event['event'] != fail_on
>>               if (event['event'] == name) and event_match(event, match):
>>                   return event
>>               self._events.append(event)
> 
> I'd rather not put assertions directly in the QEMUMachine code, unless
> it's actually an assertion and not a test failure, because this code is
> not necessarily meant to be used exclusively for tests.

It was most simple thing to do to stop test execution not handling returned 
event..
But I think you are right, I'll rethink it.

> 
>> diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255
>> new file mode 100755
>> index 0000000000..36712689d3
>> --- /dev/null
>> +++ b/tests/qemu-iotests/255
>> @@ -0,0 +1,81 @@
>> +#!/usr/bin/env python
>> +#
>> +# Tests for temporary external snapshot when we have bitmaps.
>> +#
>> +# Copyright (c) 2019 Virtuozzo International GmbH. All rights reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> +#
>> +
>> +import iotests
>> +from iotests import qemu_img_create, file_path, log
>> +
>> +iotests.verify_image_format(supported_fmts=['qcow2'])
>> +
>> +base, top = file_path('base', 'top')
>> +size = 64 * 1024 * 3
>> +
>> +
>> +def print_bitmap(msg, vm):
>> +    result = vm.qmp('query-block')['return'][0]
>> +    if 'dirty-bitmaps' in result:
>> +        bitmap = result['dirty-bitmaps'][0]
>> +        log('{}: name={} dirty-clusters={}'.format(msg, bitmap['name'],
>> +            bitmap['count'] // 64 // 1024))
>> +    else:
>> +        log(msg + ': not found')
>> +
>> +
>> +def test(persistent, restart):
>> +    assert persistent or not restart
>> +    log("\nTestcase {}persistent {} restart\n".format(
>> +            '' if persistent else 'non-', 'with' if restart else 'without'))
>> +
>> +    qemu_img_create('-f', iotests.imgfmt, base, str(size))
>> +
>> +    vm = iotests.VM().add_drive(base)
>> +    vm.launch()
>> +
>> +    vm.qmp_log('block-dirty-bitmap-add', node='drive0', name='bitmap0',
>> +               persistent=persistent)
>> +    vm.hmp_qemu_io('drive0', 'write 0 64K')
>> +    print_bitmap('initial bitmap', vm)
>> +
>> +    vm.qmp_log('blockdev-snapshot-sync', device='drive0', snapshot_file=top,
>> +               format=iotests.imgfmt, 
>> filters=[iotests.filter_qmp_testfiles])
>> +    vm.hmp_qemu_io('drive0', 'write 64K 512')
>> +    print_bitmap('check, that no bitmaps in snapshot', vm)
> 
> 'check that no bitmaps are in snapshot', probably.
> 
>> +
>> +    if restart:
>> +        log("... Restart ...")
>> +        vm.shutdown()
>> +        vm = iotests.VM().add_drive(top)
>> +        vm.launch()
>> +
>> +    vm.qmp_log('block-commit', device='drive0', top=top,
>> +               filters=[iotests.filter_qmp_testfiles])
>> +    log(vm.event_wait('BLOCK_JOB_READY', fail_on='BLOCK_JOB_COMPLETED'),
>> +        filters=[iotests.filter_qmp_event])
> 
> Looks like you probably saw some interesting things during development.
> You shouldn't see COMPLETED before READY, should you? Is this necessary?

When job failed, we may not have READY event but only COMPLETED with failure.

I think, I can instead improve event_wait to support list of events to wait,
and than check in test, what is it, READY or COMPLETED.

> 
>> +    vm.qmp_log('block-job-complete', device='drive0')
>> +    log(vm.event_wait('BLOCK_JOB_COMPLETED'),
>> +        filters=[iotests.filter_qmp_event])
> 
> (Just musing: we probably want a filtered version of event_wait for
> iotests.py so we don't have to type this so much.)

agree

> 
>> +    print_bitmap('check merged bitmap', vm)
>> +
> 
> Merged? We don't /merge/ the bitmaps, do we? I guess you mean to say
> something like: "Check bitmaps on merged/converged node", right?

Yes, something like this.

> 
>> +    vm.hmp_qemu_io('drive0', 'write 128K 64K')
>> +    print_bitmap('check updated bitmap', vm)
>> +
>> +    vm.shutdown()
>> +
>> +
>> +test(persistent=False, restart=False)
>> diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
>> new file mode 100644
>> index 0000000000..2bffb486d2
>> --- /dev/null
>> +++ b/tests/qemu-iotests/255.out
>> @@ -0,0 +1,17 @@
>> +
>> +Testcase non-persistent without restart
>> +
>> +{"execute": "block-dirty-bitmap-add", "arguments": {"name": "bitmap0", 
>> "node": "drive0", "persistent": false}}
>> +{"return": {}}
>> +initial bitmap: name=bitmap0 dirty-clusters=1
>> +{"execute": "blockdev-snapshot-sync", "arguments": {"device": "drive0", 
>> "format": "qcow2", "snapshot-file": "TEST_DIR/PID-top"}}
>> +{"return": {}}
>> +check, that no bitmaps in snapshot: not found
>> +{"execute": "block-commit", "arguments": {"device": "drive0", "top": 
>> "TEST_DIR/PID-top"}}
>> +{"return": {}}
>> +{"data": {"device": "drive0", "len": 65536, "offset": 65536, "speed": 0, 
>> "type": "commit"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": 
>> "USECS", "seconds": "SECS"}}
>> +{"execute": "block-job-complete", "arguments": {"device": "drive0"}}
>> +{"return": {}}
>> +{"data": {"device": "drive0", "len": 65536, "offset": 65536, "speed": 0, 
>> "type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": 
>> {"microseconds": "USECS", "seconds": "SECS"}}
>> +check merged bitmap: name=bitmap0 dirty-clusters=2
>> +check updated bitmap: name=bitmap0 dirty-clusters=3
>> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
>> index 52b7c16e15..2758f48143 100644
>> --- a/tests/qemu-iotests/group
>> +++ b/tests/qemu-iotests/group
>> @@ -251,3 +251,4 @@
>>   249 rw auto quick
>>   252 rw auto backing quick
>>   253 rw auto quick
>> +255 rw auto quick
>>
> 
> Good test, I just have to dig through patch 3.
> 


-- 
Best regards,
Vladimir