Re: [Qemu-devel] [Qemu-block] [PATCH v1] [RFC] qcow2: add compression type feature

[John Snow]

On 5/17/19 4:05 AM, Denis Plotnikov wrote:
> 
> 
> On 17.05.2019 2:25, John Snow wrote:
>>
>>
>> On 5/16/19 9:48 AM, Denis Plotnikov wrote:
>>> The patch adds some preparation parts for incompatible compression type
>>> feature into QCOW2 header that indicates that *all* compressed clusters
>>> must be (de)compressed using a certain compression type.
>>>
>>> It is implied that the compression type is set on the image creation and
>>> can be changed only later by image conversion, thus the only compression
>>> algorithm is used for the image.
>>>
>>> The plan is to add support for ZSTD and then may be something more effective
>>> in the future.
>>>
>>> ZSTD compression algorithm consumes 3-5 times less CPU power with a
>>> comparable compression ratio with zlib. It would be wise to use it for
>>> data compression e.g. for backups.
>>>
>>> The default compression is ZLIB.
>>>
>>
>> (Merely a curiosity:)
>>
>> Since this is coming from Virtuozzo, I trust that you've had good luck
>> with ZSTD already in R&D. What do the compression ratios look like in
>> practice? It's touted as "comparable to zlib" which certainly does sound
>> quite nice for streaming compression of backups.
>>
>> I suppose in the worst case it ought to be faster than bandwidth speeds,
>> so no harm in utilizing it.
> Yes, we did some research on it. Actually, there is a patch in the 
> mailing list (please, take a look: ) which applies ZSTD compression in 
> the migration.
> Here are the numbers from there:
> 
> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
> migration to the same host
> VM: 2xVCPU, 8G RAM total
> 5G RAM used, memory populated with postgreqsl data
> produced by pgbench performance benchmark
> 
> Threads: 1 compress – 1 decompress
> 
> zstd provides slightly less compression ratio with almost the same
> CPU usage but copes with RAM compression roughly 2 times faster
> 
> compression type              zlib       |      zstd
> ---------------------------------------------------------
> compression level          1       5     |   1       5
> compression ratio          6.92    7.05  |   6.69    6.89
> cpu idle, %                82      83    |   86      80
> time, sec                  49      71    |   26      31
> time diff to zlib, sec                   |  -25     -41
> time diff to zlib, %                     |  -47%    -56%
> 
> I general ZSTD provides better compression ratio on big dependent chunks 
> of data. Than bigger the data size then better ZSTD it compresses.
> 
> Since, in our cases (migration: 4K RAM block, qcow2: cluster) we
> have to compress independent chunks the ability of ZSTD to find better 
> compression solution is restricted.
> 
> Although, the compression ratio is pretty much the same in both cases 
> the experiments shown that ZSTD does the compression much faster (x2).
> 
> Which is obviously good for us.
> 
> Here is other comparison of ZLIB vs ZSTD without any application in qemu 
> which shows that ZSTD works faster
> 
> zlib = max compression level
> zstd = compression level 5 (max 22)
> 
> cycles consumed for compression:
>                                               
>       4k txt  4K ram  4K bin  64K txt  64K ram  64K bin
> zlib    400K    344K    1.3M      13M       5M    92.3M
> zstd    350K    235K    312K     3.3M     1.3M     2.4M
> Diff,%           -12     -32     -77      -75      -73      -97
> 

Wow, the 4k bin one is drastic. The text is even more prominent. wow!
                                                
> size after compression in bytes:
> 
>       4k txt  4K ram  4K bin  64K txt  64K ram  64K bin
> zlib    1542    3599    1403  17386     64735     20609
> zstd    1568    3540    1250  17656     65546     20023
> Diff,%            -2       2      11     -2        -1         3
> 

Yeah, that's pretty close. Seems like absolutely a great tradeoff for
the speed gain. If the little bit of difference matters to you, you can
always do some more heavy-duty compression of your choice in another
layer of the storage stack.

> Data sources for test files [we took 4K and 64K chunks from there]:
> 
> txt = linux/Documentation/memory-barriers.txt
> ram = /boot/initramfs-4.20.0-rc6+.img
> bin = x86_64-softmmu/qemu-system-x86_64
> 
> Increasing of ZSTD compression ratio didn't give any significant 
> improvements of the out size but slowed down the pace of ZSTD
> 
> Denis
> 
>>
>>> Signed-off-by: Denis Plotnikov <address@hidden>
> 

Very useful data, thanks for sharing! Seems like this would indeed be a
great thing to have for qcow2.