[Qemu-commits] [qemu/qemu] 3c9503: qapi/block-core: Introduce BackupCommon

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 3c95037aa857e95916b2a91e127ec9b74af8ed05
      
https://github.com/qemu/qemu/commit/3c95037aa857e95916b2a91e127ec9b74af8ed05
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M qapi/block-core.json

  Log Message:
  -----------
  qapi/block-core: Introduce BackupCommon

drive-backup and blockdev-backup have an awful lot of things in common
that are the same. Let's fix that.

I don't deduplicate 'target', because the semantics actually did change
between each structure. Leave that one alone so it can be documented
separately.

Where documentation was not identical, use the most up-to-date version.
For "speed", use Blockdev-Backup's version. For "sync", use
Drive-Backup's version.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
[Maintainer edit: modified commit message. --js]
Reviewed-by: Markus Armbruster <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 7b0b870bccc1b663519ef8f56ad2c67fc3530fe9
      
https://github.com/qemu/qemu/commit/7b0b870bccc1b663519ef8f56ad2c67fc3530fe9
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M blockdev.c

  Log Message:
  -----------
  drive-backup: create do_backup_common

Create a common core that comprises the actual meat of what the backup API
boundary needs to do, and then switch drive-backup to use it.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 920305661473842980b65fca439af2bb69fcec76
      
https://github.com/qemu/qemu/commit/920305661473842980b65fca439af2bb69fcec76
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M blockdev.c

  Log Message:
  -----------
  blockdev-backup: utilize do_backup_common

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 00a463b1dc73d1665ce6720df4de052aff95acf8
      
https://github.com/qemu/qemu/commit/00a463b1dc73d1665ce6720df4de052aff95acf8
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M qapi/block-core.json

  Log Message:
  -----------
  qapi: add BitmapSyncMode enum

Depending on what a user is trying to accomplish, there might be a few
bitmap cleanup actions that occur when an operation is finished that
could be useful.

I am proposing three:
- NEVER: The bitmap is never synchronized against what was copied.
- ALWAYS: The bitmap is always synchronized, even on failures.
- ON-SUCCESS: The bitmap is synchronized only on success.

The existing incremental backup modes use 'on-success' semantics,
so add just that one for right now.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Reviewed-by: Markus Armbruster <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: c8b56501780615f709d96419fd718ca562d41c46
      
https://github.com/qemu/qemu/commit/c8b56501780615f709d96419fd718ca562d41c46
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M block/mirror.c
    M block/replication.c
    M blockdev.c
    M include/block/block_int.h
    M qapi/block-core.json

  Log Message:
  -----------
  block/backup: Add mirror sync mode 'bitmap'

We don't need or want a new sync mode for simple differences in
semantics.  Create a new mode simply named "BITMAP" that is designed to
make use of the new Bitmap Sync Mode field.

Because the only bitmap sync mode is 'on-success', this adds no new
functionality to the backup job (yet). The old incremental backup mode
is maintained as a syntactic sugar for sync=bitmap, mode=on-success.

Add all of the plumbing necessary to support this new instruction.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: cf0cd293c6311e0714739a95d022b262e1d9d798
      
https://github.com/qemu/qemu/commit/cf0cd293c6311e0714739a95d022b262e1d9d798
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M qapi/block-core.json

  Log Message:
  -----------
  block/backup: add 'never' policy to bitmap sync mode

This adds a "never" policy for bitmap synchronization. Regardless of if
the job succeeds or fails, we never update the bitmap. This can be used
to perform differential backups, or simply to avoid the job modifying a
bitmap.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 3bde4b010e7510061dd8055b336c0148610a7dff
      
https://github.com/qemu/qemu/commit/3bde4b010e7510061dd8055b336c0148610a7dff
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M util/hbitmap.c

  Log Message:
  -----------
  hbitmap: Fix merge when b is empty, and result is not an alias of a

Nobody calls the function like this currently, but we neither prohibit
or cope with this behavior. I decided to make the function cope with it.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: c5b40c1f9cd310b44b571a75ee42de22539996cd
      
https://github.com/qemu/qemu/commit/c5b40c1f9cd310b44b571a75ee42de22539996cd
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M util/hbitmap.c

  Log Message:
  -----------
  hbitmap: enable merging across granularities

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: b7661ca5d84a7d330acc3c3f471e757e55261f84
      
https://github.com/qemu/qemu/commit/b7661ca5d84a7d330acc3c3f471e757e55261f84
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/dirty-bitmap.c
    M include/block/block_int.h

  Log Message:
  -----------
  block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

I'm surprised it didn't come up sooner, but sometimes we have a +busy
bitmap as a source. This is dangerous from the QMP API, but if we are
the owner that marked the bitmap busy, it's safe to merge it using it as
a read only source.

It is not safe in the general case to allow users to read from in-use
bitmaps, so create an internal variant that foregoes the safety
checking.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 28636b8211c30e9b7ab05806a8824eabf377f307
      
https://github.com/qemu/qemu/commit/28636b8211c30e9b7ab05806a8824eabf377f307
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/dirty-bitmap.c
    M block/mirror.c
    M include/block/dirty-bitmap.h
    M migration/block.c
    M nbd/server.c

  Log Message:
  -----------
  block/dirty-bitmap: add bdrv_dirty_bitmap_get

Add a public interface for get. While we're at it,
rename "bdrv_get_dirty_bitmap_locked" to "bdrv_dirty_bitmap_get_locked".

(There are more functions to rename to the bdrv_dirty_bitmap_VERB form,
but they will wait until the conclusion of this series.)

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 62aa1fbeac5922240fedd22b435024a3b96cbc8f
      
https://github.com/qemu/qemu/commit/62aa1fbeac5922240fedd22b435024a3b96cbc8f
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c

  Log Message:
  -----------
  block/backup: upgrade copy_bitmap to BdrvDirtyBitmap

This simplifies some interface matters; namely the initialization and
(later) merging the manifest back into the sync_bitmap if it was
provided.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: c23909e5301b506bc9a59d178129cfdf6b75bda7
      
https://github.com/qemu/qemu/commit/c23909e5301b506bc9a59d178129cfdf6b75bda7
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M qapi/block-core.json

  Log Message:
  -----------
  block/backup: add 'always' bitmap sync policy

This adds an "always" policy for bitmap synchronization. Regardless of if
the job succeeds or fails, the bitmap is *always* synchronized. This means
that for backups that fail part-way through, the bitmap retains a record of
which sectors need to be copied out to accomplish a new backup using the
old, partial result.

In effect, this allows us to "resume" a failed backup; however the new backup
will be from the new point in time, so it isn't a "resume" as much as it is
an "incremental retry." This can be useful in the case of extremely large
backups that fail considerably through the operation and we'd like to not waste
the work that was already performed.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 456a2d5ac7641c7e75c76328a561b528a8607a8e
      
https://github.com/qemu/qemu/commit/456a2d5ac7641c7e75c76328a561b528a8607a8e
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  iotests: add testing shim for script-style python tests

Because the new-style python tests don't use the iotests.main() test
launcher, we don't turn on the debugger logging for these scripts
when invoked via ./check -d.

Refactor the launcher shim into new and old style shims so that they
share environmental configuration.

Two cleanup notes: debug was not actually used as a global, and there
was no reason to create a class in an inner scope just to achieve
default variables; we can simply create an instance of the runner with
the values we want instead.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: d443b74b3d752640e57a0f7e8efcc09d33504e63
      
https://github.com/qemu/qemu/commit/d443b74b3d752640e57a0f7e8efcc09d33504e63
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  iotests: teach run_job to cancel pending jobs

run_job can cancel pending jobs to simulate failure. This lets us use
the pending callback to issue test commands while the job is open, but
then still have the job fail in the end.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
[Maintainer edit: Merge conflict resolution in run_job]
Signed-off-by: John Snow <address@hidden>


  Commit: de263986b5dc7571d12a95305ffc7ddd2f349431
      
https://github.com/qemu/qemu/commit/de263986b5dc7571d12a95305ffc7ddd2f349431
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  iotests: teach FilePath to produce multiple paths

Use "FilePaths" instead of "FilePath" to request multiple files be
cleaned up after we leave that object's scope.

This is not crucial; but it saves a little typing.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: f357576fa9cfc6413a9629145dbe2e56a11d7e0d
      
https://github.com/qemu/qemu/commit/f357576fa9cfc6413a9629145dbe2e56a11d7e0d
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/040
    M tests/qemu-iotests/093
    M tests/qemu-iotests/139
    M tests/qemu-iotests/238
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  iotests: Add virtio-scsi device helper

Seems that it comes up enough.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: dfdc48d5e2c48d28fd9aa8829928d77650b350ae
      
https://github.com/qemu/qemu/commit/dfdc48d5e2c48d28fd9aa8829928d77650b350ae
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    A tests/qemu-iotests/257
    A tests/qemu-iotests/257.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: add test 257 for bitmap-mode backups

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
[Removed 'auto' group, as per new testing config guidelines --js]
Signed-off-by: John Snow <address@hidden>


  Commit: b30ffbef531bb851dd96f335de4788e90471234e
      
https://github.com/qemu/qemu/commit/b30ffbef531bb851dd96f335de4788e90471234e
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M blockdev.c

  Log Message:
  -----------
  block/backup: loosen restriction on readonly bitmaps

With the "never" sync policy, we actually can utilize readonly bitmaps
now. Loosen the check at the QMP level, and tighten it based on
provided arguments down at the job creation level instead.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 2899f41eef2806cf8eb119811c9d6fcf15ce80f6
      
https://github.com/qemu/qemu/commit/2899f41eef2806cf8eb119811c9d6fcf15ce80f6
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M blockdev.c

  Log Message:
  -----------
  blockdev: reduce aio_context locked sections in bitmap add/remove

Commit 0a6c86d024c52 returned these locks back to add/remove
functionality, to protect from intersection of persistent bitmap
related IO with other IO. But other bitmap-related functions called
here are unrelated to the problem, and there are no needs to keep these
calls inside critical sections.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>
Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: c4e4b0fa598ddc9cee6ba7a06899ce0a8dae6c61
      
https://github.com/qemu/qemu/commit/c4e4b0fa598ddc9cee6ba7a06899ce0a8dae6c61
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block.c
    M block/dirty-bitmap.c
    M blockdev.c
    M include/block/dirty-bitmap.h
    M migration/block-dirty-bitmap.c
    M qapi/transaction.json

  Log Message:
  -----------
  qapi: implement block-dirty-bitmap-remove transaction action

It is used to do transactional movement of the bitmap (which is
possible in conjunction with merge command). Transactional bitmap
movement is needed in scenarios with external snapshot, when we don't
want to leave copy of the bitmap in the base image.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
[Edited "since" version to 4.2 --js]
Signed-off-by: John Snow <address@hidden>


  Commit: 3f7b2fa8cd476fe871ce1d996c640317730752a0
      
https://github.com/qemu/qemu/commit/3f7b2fa8cd476fe871ce1d996c640317730752a0
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/254
    M tests/qemu-iotests/254.out

  Log Message:
  -----------
  iotests: test bitmap moving inside 254

Test persistent bitmap copying with and without removal of original
bitmap.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
[Edited comment "bitmap1" --> "bitmap2" as per review. --js]
Signed-off-by: John Snow <address@hidden>


  Commit: b0a32bef7bd9f70b1c1dd841934b67d0e9e6f0c0
      
https://github.com/qemu/qemu/commit/b0a32bef7bd9f70b1c1dd841934b67d0e9e6f0c0
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/257

  Log Message:
  -----------
  iotests/257: add Pattern class

Just kidding, this is easier to manage with a full class instead of a
namedtuple.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 32afa5a1d4f2701a4fe7b67790b856caad22fe0c
      
https://github.com/qemu/qemu/commit/32afa5a1d4f2701a4fe7b67790b856caad22fe0c
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/257

  Log Message:
  -----------
  iotests/257: add EmulatedBitmap class

Represent a bitmap with an object that we can mark and clear bits in.
This makes it easier to manage partial writes when we don't write a
full group's worth of patterns before an error.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 0af2a09c6baac5c9ee93b6e78dcc6bb029361620
      
https://github.com/qemu/qemu/commit/0af2a09c6baac5c9ee93b6e78dcc6bb029361620
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/257
    M tests/qemu-iotests/257.out

  Log Message:
  -----------
  iotests/257: Refactor backup helpers

This test needs support for non-bitmap backups and missing or
unspecified bitmap sync modes, so rewrite the helpers to be a little
more generic.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: a6c9365ad4ba26b274bb4b7491f99b05bb765d76
      
https://github.com/qemu/qemu/commit/a6c9365ad4ba26b274bb4b7491f99b05bb765d76
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M blockdev.c

  Log Message:
  -----------
  block/backup: hoist bitmap check into QMP interface

This is nicer to do in the unified QMP interface that we have now,
because it lets us use the right terminology back at the user.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 352092d3828adb67b28d30588e3c4635234e2113
      
https://github.com/qemu/qemu/commit/352092d3828adb67b28d30588e3c4635234e2113
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/257
    M tests/qemu-iotests/257.out

  Log Message:
  -----------
  iotests/257: test API failures

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 0fff1f13718a80c12a26dfaee17bdeb45ce51459
      
https://github.com/qemu/qemu/commit/0fff1f13718a80c12a26dfaee17bdeb45ce51459
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M tests/qemu-iotests/256.out
    M tests/qemu-iotests/257.out

  Log Message:
  -----------
  block/backup: improve sync=bitmap work estimates

When making backups based on bitmaps, the work estimate can be more
accurate. Update iotests to reflect the new strategy.

TOP work estimates are broken, but do not get worse with this commit.
That issue is addressed in the following commits instead.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 141cdcdf84a07c8f23f8b1c4e3c5ebb0669f7616
      
https://github.com/qemu/qemu/commit/141cdcdf84a07c8f23f8b1c4e3c5ebb0669f7616
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c

  Log Message:
  -----------
  block/backup: centralize copy_bitmap initialization

Just a few housekeeping changes that keeps the following commit easier
to read; perform the initial copy_bitmap initialization in one place.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: dba8700f16ebda0632977c303f66021407971081
      
https://github.com/qemu/qemu/commit/dba8700f16ebda0632977c303f66021407971081
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c

  Log Message:
  -----------
  block/backup: add backup_is_cluster_allocated

Modify bdrv_is_unallocated_range to utilize the pnum return from
bdrv_is_allocated, and in the process change the semantics from
"is unallocated" to "is allocated."

Optionally returns a full number of clusters that share the same
allocation status.

This will be used to carefully toggle bits in the bitmap for sync=top
initialization in the following commits.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 7e30dd618ebfe3ab1ed54f2e98ba75d799c0be20
      
https://github.com/qemu/qemu/commit/7e30dd618ebfe3ab1ed54f2e98ba75d799c0be20
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M block/trace-events

  Log Message:
  -----------
  block/backup: teach TOP to never copy unallocated regions

Presently, If sync=TOP is selected, we mark the entire bitmap as dirty.
In the write notifier handler, we dutifully copy out such regions.

Fix this in three parts:

1. Mark the bitmap as being initialized before the first yield.
2. After the first yield but before the backup loop, interrogate the
allocation status asynchronously and initialize the bitmap.
3. Teach the write notifier to interrogate allocation status if it is
invoked during bitmap initialization.

As an effect of this patch, the job progress for TOP backups
now behaves like this:

- total progress starts at bdrv_length.
- As allocation status is interrogated, total progress decreases.
- As blocks are copied, current progress increases.

Taken together, the floor and ceiling move to meet each other.


Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
[Remove ret = -ECANCELED change. --js]
[Squash in conflict resolution based on Max's patch --js]
Message-id: address@hidden
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: John Snow <address@hidden>


  Commit: 1a2b8b406bea1108c990b6fc932bef63211de245
      
https://github.com/qemu/qemu/commit/1a2b8b406bea1108c990b6fc932bef63211de245
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M blockdev.c
    M qapi/block-core.json

  Log Message:
  -----------
  block/backup: support bitmap sync modes for non-bitmap backups

Accept bitmaps and sync policies for the other backup modes.
This allows us to do things like create a bitmap synced to a full backup
without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything,
  because the bitmap isn't used for input or output.
  It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this
  job never completes with success anyway, and the resulting artifact
  of 'always' is suspect: because we start with a full bitmap and only
  copy out segments that get written to, the final output bitmap will
  always be ... a fully set bitmap.

  Maybe there's contexts in which bitmaps make sense for sync=none,
  but not without more severe changes to the current job, and omitting
  it here doesn't prevent us from adding it later.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: bd5ceebf6d77e680058e2325765bdedc9292ed76
      
https://github.com/qemu/qemu/commit/bd5ceebf6d77e680058e2325765bdedc9292ed76
  Author: John Snow <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/qemu-iotests/257
    M tests/qemu-iotests/257.out

  Log Message:
  -----------
  iotests/257: test traditional sync modes

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
[Edit 'Bitmap' --> 'bitmap' in 257.out --js]
Signed-off-by: John Snow <address@hidden>


  Commit: 590a63d5983c9cdf2e1bf96a519532949963085d
      
https://github.com/qemu/qemu/commit/590a63d5983c9cdf2e1bf96a519532949963085d
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/qapi.c
    M qapi/block-core.json
    M qemu-deprecated.texi

  Log Message:
  -----------
  qapi: add dirty-bitmaps to query-named-block-nodes result

Let's add a possibility to query dirty-bitmaps not only on root nodes.
It is useful when dealing both with snapshots and incremental backups.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
[Added deprecation information. --js]
Signed-off-by: John Snow <address@hidden>
[Fixed spelling --js]


  Commit: 319bd5edb92addc725b2701d73381c415146c287
      
https://github.com/qemu/qemu/commit/319bd5edb92addc725b2701d73381c415146c287
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c
    M blockdev.c

  Log Message:
  -----------
  block/backup: deal with zero detection

We have detect_zeroes option, so at least for blockdev-backup user
should define it if zero-detection is needed. For drive-backup leave
detection enabled by default but do it through existing option instead
of open-coding.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: a1ed82b4438a7fdf470673c575e8b47d276c9030
      
https://github.com/qemu/qemu/commit/a1ed82b4438a7fdf470673c575e8b47d276c9030
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M block/backup.c

  Log Message:
  -----------
  block/backup: refactor write_flags

write flags are constant, let's store it in BackupBlockJob instead of
recalculating. It also makes two boolean fields to be unused, so,
drop them.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: a5f8a60b3eafd5563af48546d5d126d448e62ac5
      
https://github.com/qemu/qemu/commit/a5f8a60b3eafd5563af48546d5d126d448e62ac5
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-08-16 (Fri, 16 Aug 2019)

  Changed paths:
    M tests/test-hbitmap.c

  Log Message:
  -----------
  tests/test-hbitmap: test next_zero and _next_dirty_area after truncate

Test that hbitmap_next_zero and hbitmap_next_dirty_area can find things
after old bitmap end.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Tested-by: John Snow <address@hidden>
Reviewed-by: John Snow <address@hidden>
Signed-off-by: John Snow <address@hidden>


  Commit: 1f37316238d0d412cbc16482c5c24b11c2c7dcec
      
https://github.com/qemu/qemu/commit/1f37316238d0d412cbc16482c5c24b11c2c7dcec
  Author: Peter Maydell <address@hidden>
  Date:   2019-08-19 (Mon, 19 Aug 2019)

  Changed paths:
    M block.c
    M block/backup.c
    M block/dirty-bitmap.c
    M block/mirror.c
    M block/qapi.c
    M block/replication.c
    M block/trace-events
    M blockdev.c
    M include/block/block_int.h
    M include/block/dirty-bitmap.h
    M migration/block-dirty-bitmap.c
    M migration/block.c
    M nbd/server.c
    M qapi/block-core.json
    M qapi/transaction.json
    M qemu-deprecated.texi
    M tests/qemu-iotests/040
    M tests/qemu-iotests/093
    M tests/qemu-iotests/139
    M tests/qemu-iotests/238
    M tests/qemu-iotests/254
    M tests/qemu-iotests/254.out
    M tests/qemu-iotests/256.out
    A tests/qemu-iotests/257
    A tests/qemu-iotests/257.out
    M tests/qemu-iotests/group
    M tests/qemu-iotests/iotests.py
    M tests/test-hbitmap.c
    M util/hbitmap.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into 
staging

Pull request

Rebase notes:

011/36:[0003] [FC] 'block/backup: upgrade copy_bitmap to BdrvDirtyBitmap'
016/36:[----] [-C] 'iotests: Add virtio-scsi device helper'
017/36:[0002] [FC] 'iotests: add test 257 for bitmap-mode backups'
030/36:[0011] [FC] 'block/backup: teach TOP to never copy unallocated regions'
032/36:[0018] [FC] 'iotests/257: test traditional sync modes'

11: A new hbitmap call was added late in 4.1, changed to
    bdrv_dirty_bitmap_next_zero.
16: Context-only (self.has_quit is new context in 040)
17: Removed 'auto' to follow upstream trends in iotest fashion
30: Handled explicitly on-list with R-B from Max.
32: Fix capitalization in test, as mentioned on-list.

# gpg: Signature made Sat 17 Aug 2019 00:12:13 BST
# gpg:                using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <address@hidden>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F  18F2 88A9 064D 1835 61EB
#      Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76  CBD0 7DEF 8106 AAFC 390E

* remotes/jnsnow/tags/bitmaps-pull-request: (36 commits)
  tests/test-hbitmap: test next_zero and _next_dirty_area after truncate
  block/backup: refactor write_flags
  block/backup: deal with zero detection
  qapi: add dirty-bitmaps to query-named-block-nodes result
  iotests/257: test traditional sync modes
  block/backup: support bitmap sync modes for non-bitmap backups
  block/backup: teach TOP to never copy unallocated regions
  block/backup: add backup_is_cluster_allocated
  block/backup: centralize copy_bitmap initialization
  block/backup: improve sync=bitmap work estimates
  iotests/257: test API failures
  block/backup: hoist bitmap check into QMP interface
  iotests/257: Refactor backup helpers
  iotests/257: add EmulatedBitmap class
  iotests/257: add Pattern class
  iotests: test bitmap moving inside 254
  qapi: implement block-dirty-bitmap-remove transaction action
  blockdev: reduce aio_context locked sections in bitmap add/remove
  block/backup: loosen restriction on readonly bitmaps
  iotests: add test 257 for bitmap-mode backups
  ...

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/afd760539308...1f37316238d0