Re: [Qemu-discuss] Discard disk image holes for sparse shrink

[Friedrich Oslage]

1. You need to set the discard option for the -drive parameter, eg 
-drive if=none,...,format=raw,discard=unmap since the default is still 
to simply ignore discards.

You may also want to set the detect-zeroes option to unmap, to discard 
all-zero blocks instead of actually writing them.


2. The disk image you created is thick-provisioned. How is qemu supposed 
to discard anything in that file? It could turn it into a sparse file or 
overwrite it with zeroes I suppose, but both options are undesirable.

To make use of the discard operation your backing storage has to support 
it. Either by beeing thin-provisioned, like qcow2 or raw sparse files, 
or by actually beeing a ssd/nvme disk.


3. Your testing method is flawed. You are using a journaling filesystem 
to write the data but to read it you access the block device directly . 
Even if the discard operation was working strings(1) would still show 
the hello-world string in the journal.

You could try something like this:

$ echo hello, world > /dev/sda
$ strings /dev/sda
$ blkdiscard /dev/sda
$ strings /dev/sda


Regards
Friedrich

On 4/9/19 4:38 PM, Pascal wrote:
> the blocks do not seem to be discarded on a raw format disk: the data
> remains on the disk...
>
> on host :
>
> dd if=/dev/zero of=/tmp/ssd.disk bs=1M count=128
> qemu -drive media=disk,file=linux.disk -device virtio-scsi-pci,id=scsi
> -device scsi-hd,drive=hd -drive if=none,id=hd,file=/tmp/ssd.disk,format=raw
>
> on Linux VM :
>
> fdisk /dev/sda
>      # create one partition on gpt type...
> fdisk -l /dev/sda
>      Disk /dev/sda: 128 MiB, 134217728 bytes, 262144 sectors
>      Units: sectors of 1 * 512 = 512 bytes
>      Sector size (logical/physical): 512 bytes / 512 bytes
>      I/O size (minimum/optimal): 512 bytes / 512 bytes
>      Disklabel type: gpt
>      Disk identifier: B8D98B8E-7790-4043-8F37-D4D8CA218884
>      Device         Start    End Sectors  Size Type
>      /dev/sda1  2048 262110  260063  127M Linux filesystem
> mkfs.ext4 -L ssd /dev/sda1
> mount /dev/sda1 /ssd
> mount
>      /dev/sda1 on /ssd type ext4 (rw,relatime,data=ordered)
> echo hello world > /ssd/test
> cat /ssd/test
>      hello world
> sync && sysctl -q vm.drop_caches=3
> rm -f /ssd/test
> sync && sysctl -q vm.drop_caches=3
> fstrim /ssd
> umount /ssd
> strings /dev/sda1 | grep 'hello world'
>      hello world
>
>
> Le mar. 9 avr. 2019 à 08:36, Narcis Garcia <address@hidden> a écrit :
>
> > Thank you.
> >
> > * I use RAW images because of it's easier to offline mount/chroot when
> > some issue happens.
> > * I call directly qemu instead of libvirt
> >
> > I'll try to parse these parameters to a qemu-system call.
> >
> >
> > El 8/4/19 a les 22:03, Friedrich Oslage ha escrit:
> > > Yes.
> > >
> > > You'll have to use the virtio-scsi driver, to my knowledge it's the only
> > > driver that supports block discards.
> > >
> > > Quick example:
> > >
> > > # create a new empty disk
> > > $ qemu-img create -f qcow2 disk.qcow2 10G
> > >
> > > # check size, should only be a few kilobytes
> > > $ du -sh disk.qcow2
> > >
> > > # add it to your vm
> > > $ virtsh edit your-vm
> > >    <disk type='block' device='disk'>
> > >      <driver name='qemu' type='qcow2' discard='unmap'
> > > detect_zeroes='unmap'/>
> > >      <source dev='/tmp/disk.qcow2'/>
> > >      <target dev='sda' bus='scsi'/>
> > >      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > >    </disk>
> > >    <controller type='scsi' index='0' model='virtio-scsi'>
> > >      <driver queues='4'/>
> > >    </controller>
> > >
> > > # in your vm, format and mount it
> > > $ mkfs.ext4 /dev/sda
> > > $ mount /dev/sda /mnt/discardtest
> > >
> > > # put some data on it
> > > $ dd if=/dev/urandom of=/mnt/discardtest/dummy.bin bs=1M count=1000
> > > $ sync
> > >
> > > # on host, check size...it should be about 1GB
> > > $ du -sh disk.qcow2
> > >
> > > # in vm, delete file and trim it
> > > $ rm /mnt/discardtest/dummy.bin
> > > $ sync
> > > $ fstrim /mnt/discardtest
> > >
> > > # on host, check size...should only be a few megabytes
> > > $ du -sh disk.qcow2
> > >
> > >
> > > It works with other guest OSes as well, for Windows you just use the
> > > Optimize-Volume cmdlets instead of fstrim.
> > >
> > > You also don't have to use qcow2 for backing. Anything that can handle
> > > discards will do, including sparse files.
> > >
> > >
> > > My advice, however, would be to use qcow2, since sparse files don't work
> > > particularly well. Executing the example above would leave you with
> > > about 300MB at the end instead of less than 10MB.
> > >
> > > Regards
> > > Friedrich
> > >
> > > On 4/5/19 4:04 PM, Narcis Garcia wrote:
> > > > No; I want to study the possibility of recovering allocated blocs of
> > > > host filesystem when guest filesystem removes its files (unallocates its
> > > > blocks).
> > > > Host -> HD or SSD (independent) with sparse-mode image
> > > > Guest -> Virtual SSD (to signal discards)
> > > >
> > > >
> > > > El 5/4/19 a les 16:01, Pascal ha escrit:
> > > > > hello,
> > > > >
> > > > > if I understand correctly, you want to study the possibility of
> > > > > recovering deleted files from an SSD disk: is that right ?
> > > > >
> > > > > regards, lacsaP.
> > > > >
> > > > > Le jeu. 4 avr. 2019 à 08:24, Narcis Garcia <address@hidden
> > > > > <mailto:address@hidden>> a écrit :
> > > > >
> > > > >       Is there a way to specify a disk to be detected as an SSD drive?
> > > > >
> > > > >       Once reached this, I want to look for the possibility to recover
> > > > > host
> > > > >       space when a guest discards disk image blocks, and this image is
> > > > > RAW
> > > > >       format and sparse allocated file.
> > > > >
> > > > >       Thank you.