bug#22517: dd byte count report does not correlate with df byte count report

[Bernhard Voelker]

On 02/01/2016 05:53 AM, Bob Gustafson wrote:
> Description of problem:
> 
> I am trying to copy /root, /home from a single disk (running the os) to a 
> bigger raid 1 disk pair running in the same chassis.
> 
> fedora is the lvm vg_group of the running system, fedora23 is the lvm 
> vg_group of the new disks (raid 1, 2TB disks).
> 
> address@hidden user1]# lvscan
>    ACTIVE            '/dev/fedora/swap' [15.57 GiB] inherit
>    ACTIVE            '/dev/fedora/home' [865.45 GiB] inherit
>    ACTIVE            '/dev/fedora/root' [50.00 GiB] inherit
>    ACTIVE            '/dev/fedora23/swap' [15.60 GiB] inherit
>    ACTIVE            '/dev/fedora23/root' [50.00 GiB] inherit
>    ACTIVE            '/dev/fedora23/home' [1.75 TiB] inherit
> 
> 
> Below is the command I'm using to copy over one partition to the raid pair:
> 
> address@hidden user1]# dd if=/dev/fedora/root of=/dev/fedora23/root
> 104857600+0 records in
> 104857600+0 records out
> 53687091200 bytes (54 GB) copied, 5639.3 s, 9.5 MB/s
> address@hidden user1]#
> 
> 
> address@hidden user1]# df -h
> Filesystem               Size  Used Avail Use% Mounted on
> devtmpfs                  16G     0   16G   0% /dev
> tmpfs                     16G  392K   16G   1% /dev/shm
> tmpfs                     16G  1.8M   16G   1% /run
> tmpfs                     16G     0   16G   0% /sys/fs/cgroup
> /dev/mapper/fedora-root   50G   27G   20G  58% /
> tmpfs                     16G   84K   16G   1% /tmp
> /dev/loop0               1.9G  3.0M  1.7G   1% /srv/node/swift_loopback
> /dev/mapper/fedora-home  852G   57G  752G   7% /home
> /dev/sdc1                477M  215M  233M  48% /boot
> tmpfs                    3.2G  8.0K  3.2G   1% /run/user/42
> tmpfs                    3.2G   24K  3.2G   1% /run/user/1000
> 
> Note that dd says 54GB were copied over into a disk partition of size only 
> 50GB. No errors during the copy. And the results show that only 27G was in 
> the fedora partition to start with. (54GB is double 27GB - this may be a clue)
> 
> Another example:
> 
> address@hidden user1]# dd if=/dev/fedora/home of=/dev/fedora23/home
> 1814978560+0 records in
> 1814978560+0 records out
> 929269022720 bytes (929 GB) copied, 32437.8 s, 28.6 MB/s
> address@hidden user1]#
> 
> Taking a look at the data size on the target partitions:
> 
> address@hidden user1]# mount /dev/fedora23/root /mnt/rootn
> address@hidden user1]# mount /dev/fedora23/home /mnt/homen
> 
> address@hidden user1]# du -h /mnt/rootn | tail -1
> 26G       /mnt/rootn
> 
> address@hidden user1]# du -h /mnt/homen | tail -1
> 57G       /mnt/homen
> 
> for the rootn partition, the 26G copied size is consistent with the source 
> size.
> 
> For the homen partition, the 57G is consistent with the source size.
> 
> However, the count given by the dd copies don't match:
> 
>   In the case of the root->rootn copy it is 54G instead of 27G
>   in the case of the home->homen copy it is 942G instead of 57G
> 
> 
> Version-Release number of selected component (if applicable):
> 
> dd is (coreutils) 8.24
> 
> How reproducible:
> 
> I did two tests - as shown
> 
> Steps to Reproduce:
> 1. create raid 1 pair
> 2. copy a 3rd single disk partition to a raid1 pair lvm partition
> 3. do df and compare results.
> 
> Actual results:
> 
> As shown above in problem narative.
> 
> Expected results:
> 
> I would think that the sizes reported by dd and df would be comparable, but 
> they are not.
> 
> Additional info:
> 
> See also https://bugzilla.redhat.com/show_bug.cgi?id=1303456

I think there are several misunderstandings here ... which is like
comparing apples and oranges.

a) the MiB numbers are 1024-based, while MB are 1000-based.

b) df(1) and du(1) show statistics or summaries of the data *inside*
a file system, while the file system itself may need some space to
manage the data.  And even df/du report different sizes:
http://www.gnu.org/software/coreutils/faq/coreutils-faq.html#df-and-du-report-different-information
Starting with BTRFS, the situation may even be worse (df showing
lots of free space while BTRFS doesn't allow to create - or even
delete - another file).

Thus said: the size of the underlying device is not 100% related to
the sizes inside the file system on that device.
You may get the real size using e.g.

  $ wc -c < /dev/sda1
  2147483648

and dd(1) will report the same size:

  $ dd if=/dev/sda1 of=/dev/null
  4194304+0 records in
  4194304+0 records out
  2147483648 bytes (2.1 GB) copied, 1.54335 s, 1.4 GB/s

Have a nice day,
Berny