[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH 1/3] qemu-img: Add checksum command
|
From: |
Hanna Reitz |
|
Subject: |
Re: [PATCH 1/3] qemu-img: Add checksum command |
|
Date: |
Wed, 26 Oct 2022 15:00:08 +0200 |
|
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 |
On 01.09.22 16:32, Nir Soffer wrote:
The checksum command compute a checksum for disk image content using the
blkhash library[1]. The blkhash library is not packaged yet, but it is
available via copr[2].
Example run:
$ ./qemu-img checksum -p fedora-35.qcow2
6e5c00c995056319d52395f8d91c7f84725ae3da69ffcba4de4c7d22cff713a5
fedora-35.qcow2
The block checksum is constructed by splitting the image to fixed sized
blocks and computing a digest of every block. The image checksum is the
digest of the all block digests.
The checksum uses internally the "sha256" algorithm but it cannot be
compared with checksums created by other tools such as `sha256sum`.
The blkhash library supports sparse images, zero detection, and
optimizes zero block hashing (they are practically free). The library
uses multiple threads to speed up the computation.
Comparing to `sha256sum`, `qemu-img checksum` is 3.5-4800[3] times
faster, depending on the amount of data in the image:
$ ./qemu-img info /scratch/50p.raw
file format: raw
virtual size: 6 GiB (6442450944 bytes)
disk size: 2.91 GiB
$ hyperfine -w2 -r5 -p "sleep 1" "./qemu-img checksum /scratch/50p.raw" \
"sha256sum /scratch/50p.raw"
Benchmark 1: ./qemu-img checksum /scratch/50p.raw
Time (mean ± σ): 1.849 s ± 0.037 s [User: 7.764 s, System:
0.962 s]
Range (min … max): 1.813 s … 1.908 s 5 runs
Benchmark 2: sha256sum /scratch/50p.raw
Time (mean ± σ): 14.585 s ± 0.072 s [User: 13.537 s, System:
1.003 s]
Range (min … max): 14.501 s … 14.697 s 5 runs
Summary
'./qemu-img checksum /scratch/50p.raw' ran
7.89 ± 0.16 times faster than 'sha256sum /scratch/50p.raw'
The new command is available only when `blkhash` is available during
build. To test the new command please install the `blkhash-devel`
package:
$ dnf copr enable nsoffer/blkhash
$ sudo dnf install blkhash-devel
[1] https://gitlab.com/nirs/blkhash
[2] https://copr.fedorainfracloud.org/coprs/nsoffer/blkhash/
[3] Computing checksum for 8T empty image: qemu-img checksum: 3.7s,
sha256sum (estimate): 17,749s
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
---
docs/tools/qemu-img.rst | 22 +++++
meson.build | 10 ++-
meson_options.txt | 2 +
qemu-img-cmds.hx | 8 ++
qemu-img.c | 191 ++++++++++++++++++++++++++++++++++++++++
5 files changed, 232 insertions(+), 1 deletion(-)
diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
index 85a6e05b35..8be9c45cbf 100644
--- a/docs/tools/qemu-img.rst
+++ b/docs/tools/qemu-img.rst
@@ -347,20 +347,42 @@ Command description:
Check completed, image is corrupted
3
Check completed, image has leaked clusters, but is not corrupted
63
Checks are not supported by the image format
If ``-r`` is specified, exit codes representing the image state refer to the
state after (the attempt at) repairing it. That is, a successful ``-r all``
will yield the exit code 0, independently of the image state before.
+.. option:: checksum [--object OBJECTDEF] [--image-opts] [-f FMT] [-T SRC_CACHE] [-p] FILENAME
+
+ Print a checksum for image *FILENAME* guest visible content.
Why not say which kind of checksum it is?
Images with
+ different format or settings wil have the same checksum.
s/wil/will/
+
+ The format is probed unless you specify it by ``-f``.
+
+ The checksum is computed for guest visible content. Allocated areas full of
+ zeroes, zero clusters, and unallocated areas are read as zeros so they will
+ have the same checksum. Images with single or multiple files or backing files
+ will have the same checksums if the guest will see the same content when
+ reading the image.
+
+ Image metadata that is not visible to the guest such as dirty bitmaps does
+ not affect the checksum.
+
+ Computing a checksum requires a read-only image. You cannot compute a
+ checksum of an active image used by a guest,
Makes me ask: Why not? Other subcommands have the -U flag for this.
but you can compute a checksum
+ of a guest during pull mode incremental backup using NBD URL.
+
+ The checksum is not compatible with other tools such as *sha256sum*.
Why not? I can see it differs even for raw images, but why? I would
have very much assumed that this gives me exactly what sha256sum in the
guest on the guest device would yield.
+
.. option:: commit [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [-t
CACHE] [-b BASE] [-r RATE_LIMIT] [-d] [-p] FILENAME
Commit the changes recorded in *FILENAME* in its base image or backing file.
If the backing file is smaller than the snapshot, then the backing file
will be
resized to be the same size as the snapshot. If the snapshot is smaller
than
the backing file, the backing file will not be truncated. If you want the
backing file to match the size of the smaller snapshot, you can safely
truncate
it yourself once the commit operation successfully completes.
The image *FILENAME* is emptied after the operation has succeeded. If you do
[...]
diff --git a/qemu-img.c b/qemu-img.c
index 7d4b33b3da..7edcfe4bc8 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -17,20 +17,21 @@
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/
#include "qemu/osdep.h"
#include <getopt.h>
+#include <blkhash.h>
This must be guarded by CONFIG_BLKHASH.
#include "qemu/help-texts.h"
#include "qemu/qemu-progress.h"
#include "qemu-version.h"
#include "qapi/error.h"
#include "qapi/qapi-commands-block-core.h"
#include "qapi/qapi-visit-block-core.h"
#include "qapi/qobject-output-visitor.h"
#include "qapi/qmp/qjson.h"
#include "qapi/qmp/qdict.h"
@@ -1611,20 +1612,210 @@ out:
qemu_vfree(buf1);
qemu_vfree(buf2);
blk_unref(blk2);
out2:
blk_unref(blk1);
out3:
qemu_progress_end();
return ret;
}
+#ifdef CONFIG_BLKHASH
+/*
+ * Compute image checksum.
+ */
+static int img_checksum(int argc, char **argv)
+{
+ const char *digest_name = "sha256";
Why don’t we make this configurable?
+ const size_t block_size = 64 * KiB;
+
+ const char *format = NULL;
+ const char *cache = BDRV_DEFAULT_CACHE;
+ const char *filename;
+ BlockBackend *blk;
+ BlockDriverState *bs;
+ uint8_t *buf = NULL;
+ int64_t pnum;
+ bool progress = false;
+ int flags = 0;
+ bool writethrough;
+ int64_t total_size;
+ int64_t offset = 0;
+ int c;
+ bool image_opts = false;
+ struct blkhash *h = NULL;
+ unsigned char digest[64];
+ unsigned int digest_len;
+ int ret = 1;
+ int err;
+
+ for (;;) {
+ static const struct option long_options[] = {
+ {"help", no_argument, 0, 'h'},
+ {"object", required_argument, 0, OPTION_OBJECT},
+ {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
+ {0, 0, 0, 0}
+ };
+ c = getopt_long(argc, argv, ":hf:T:p",
+ long_options, NULL);
+ if (c == -1) {
+ break;
+ }
+ switch (c) {
+ case ':':
+ missing_argument(argv[optind - 1]);
+ break;
+ case '?':
+ unrecognized_option(argv[optind - 1]);
+ break;
+ case 'h':
+ help();
+ break;
+ case 'f':
+ format = optarg;
+ break;
+ case 'T':
+ cache = optarg;
+ break;
+ case 'p':
+ progress = true;
+ break;
+ case OPTION_OBJECT:
+ {
+ Error *local_err = NULL;
+
+ if (!user_creatable_add_from_str(optarg, &local_err)) {
+ if (local_err) {
+ error_report_err(local_err);
+ exit(2);
+ } else {
+ /* Help was printed */
+ exit(EXIT_SUCCESS);
+ }
+ }
How about simply using user_creatable_process_cmdline(optarg)? (which
most other subcommands use)
(I believe img_compare() has to have its own code, because exit code 1
is reserved there. But that isn’t a concern here.)
+ break;
+ }
+ case OPTION_IMAGE_OPTS:
+ image_opts = true;
+ break;
+ }
+ }
+
+ if (optind != argc - 1) {
+ error_exit("Expecting image file name");
+ }
+
+ filename = argv[optind++];
+
+ err = bdrv_parse_cache_mode(cache, &flags, &writethrough);
+ if (err < 0) {
+ error_report("Invalid source cache option: %s", cache);
+ return ret;
Personally, I’m not too big of a fan of using `ret` here, because it was
set so far above. Why not just `return 1;`?
(Same in other error cases below.)
+ }
+
+ blk = img_open(image_opts, filename, format, flags, writethrough, false,
+ false);
+ if (!blk) {
+ return ret;
+ }
+
+ /* Initialize before using goto out. */
+ qemu_progress_init(progress, 2.0);
+
+ bs = blk_bs(blk);
+ buf = blk_blockalign(blk, IO_BUF_SIZE);
It looks like this macro is kind of specific to `img_compare()`, and is
currently somewhere in the middle of its code. I don’t mind reusing it
here, but if so, we might want to move it up to the top of this file,
and add a comment that this is the buffer size used for commands like
compare or checksum.
+
+ total_size = blk_getlength(blk);
+ if (total_size < 0) {
+ error_report("Can't get size of %s: %s",
+ filename, strerror(-total_size));
+ goto out;
I suggest adding `ret = 1;` before such a `goto out;` (not just here),
so it is clear exactly what value we are going to return (it isn’t
trivial to track that `ret` isn’t used for anything else). But I can see
that it’s extra code, so maybe you don’t like that.
If we keep this as-is, perhaps we could rename `ret` to something more
explicit like `exit_status`, though? That way, it’s pretty clear that
we won’t accidentally reuse it for anything else. (I know this isn’t
the first subcommand to use `ret` for the process exist status, but I
don’t think those other subcommands are great role models in this regard.)
+ }
+
+ h = blkhash_new(block_size, digest_name);
Should we somehow make sure that IO_BUF_SIZE is a multiple of
block_size? I mean, it is, but it isn’t obvious at least, and I guess
maybe at some point someone might want to make block_size a parameter.
Would a static assertion work? (Would stop working once someone decide
to make block_size a parameter, which is good, because it draws attention.)
+ if (!h) {
+ error_report("Can't create blkhash: %s", strerror(errno));
+ goto out;
+ }
+
+ qemu_progress_print(0, 100);
+
+ while (offset < total_size) {
+ int status;
+ int64_t chunk;
+
+ status = bdrv_block_status_above(bs, NULL, offset,
+ total_size - offset, &pnum, NULL,
+ NULL);
+ if (status < 0) {
+ error_report("Error checking status at offset %" PRId64 " for %s",
+ offset, filename);
+ goto out;
+ }
+
+ assert(pnum);
+ chunk = pnum;
Can’t we drop `pnum` and replace it by `chunk` in all cases?
+
+ if (status & BDRV_BLOCK_ZERO) {
+ chunk = MIN(chunk, SIZE_MAX);
Itches me a bit to propose rounding SIZE_MAX down to block_size, but I
guess given the magnitude of SIZE_MAX on 64-bit systems, it doesn’t matter.
+ err = blkhash_zero(h, chunk);
+ if (err) {
+ error_report("Error zeroing hash at offset %" PRId64
+ " of %s: %s",
+ offset, filename, strerror(err));
+ goto out;
+ }
+ } else {
+ chunk = MIN(chunk, IO_BUF_SIZE);
+ err = blk_pread(blk, offset, chunk, buf, 0);
+ if (err < 0) {
+ error_report("Error reading at offset %" PRId64 " of %s: %s",
+ offset, filename, strerror(-err));
+ goto out;
+ }
+ err = blkhash_update(h, buf, chunk);
+ if (err) {
+ error_report("Error updating hash at offset %" PRId64
+ " of %s: %s",
+ offset, filename, strerror(err));
+ goto out;
+ }
+ }
+
+ offset += chunk;
+ qemu_progress_print(((float) chunk / total_size) * 100, 100);
+ }
+
+ err = blkhash_final(h, digest, &digest_len);
How does this verify that `digest` is sufficiently large?
I mean, it is, given that we only have sha256 now. But it still seems
rather dangerous to me.
+ if (err) {
+ error_report("Error finalizing hash of %s: %s",
+ filename, strerror(err));
+ goto out;
+ }
+
+ for (unsigned i = 0; i < digest_len; i++) {
I always such declarations weren’t allowed in qemu, but this isn’t the
first place, and I don’t mind. Good to know. :)
Hanna
+ printf("%02x", digest[i]);
+ }
+ printf(" %s%s", filename, progress ? "" : "\n");
+
+ ret = 0;
+
+out:
+ blkhash_free(h);
+ qemu_vfree(buf);
+ blk_unref(blk);
+ qemu_progress_end();
+
+ return ret;
+}
+#endif /* CONFIG_BLKHASH */
+
/* Convenience wrapper around qmp_block_dirty_bitmap_merge */
static void do_dirty_bitmap_merge(const char *dst_node, const char *dst_name,
const char *src_node, const char *src_name,
Error **errp)
{
BlockDirtyBitmapOrStr *merge_src;
BlockDirtyBitmapOrStrList *list = NULL;
merge_src = g_new0(BlockDirtyBitmapOrStr, 1);
merge_src->type = QTYPE_QDICT;
- Re: [PATCH 1/3] qemu-img: Add checksum command,
Hanna Reitz <=