--- Begin Message ---
|
Subject: |
dd reads random number of records from pipes - named or otherwise - coreutils 8.9 |
|
Date: |
Tue, 12 Apr 2011 14:02:12 -0700 |
|
User-agent: |
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 |
I can't believe such an obvious bug would exist this long, but on
the other hand the test is so simple I can't see where it's user
error.
dd, when reading from stdin or from a named pipe sometimes (but
not always) reads a random number of records a bit less then what
it should.
I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and
even mkfifo pip; yes > pipe & dd if=pipe -- and all sometimes
failed.
However, if=arealfile seems to always work perfectly.
To replicate:
yes|dd bs=1000 count=1000|wc -c
cat /dev/zero |dd bs=1000 count=1000 |wc -c
If it works perfectly the first time, just keep trying. For me,
it randomly works and doesn't work. Some conditions are more
likely to work, and others to fail.
For me, using the "yes" method above almost always reads the
incorrect number of bytes, while the /dev/zero method usually
works correctly but occasionally reads the wrong number of bytes.
The problem exists on the following coronations:
Slackware 12.0.0 (2.6.21.5-smp) / coreutils 8.9
Slackware 12.0.0 (2.6.21.5-smp) / coreutils 6.9
Slackware 13.0.0 (2.6.29.6) / coreutils 7.9
Kubuntu lenny/sid (2.6.24-27-generic SMP) / coreutils 6.10
(I only mention the older versions of coreutils because it may be
helpful. I'm only filing the bug report for 8.9!)
Thanks,
Jesse
For example:
address@hidden:~# dd --version
dd (coreutils) 8.9
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Paul Rubin, David MacKenzie, and Stuart Kemp.
address@hidden:~# yes|dd bs=1000 count=1000|wc -c
597+403 records in
597+403 records out
608232 bytes (608 kB) copied, 0.0263828 s, 23.1 MB/s
608232
address@hidden:~# yes|dd bs=1000 count=1000|wc -c
885+115 records in
885+115 records out
887784 bytes (888 kB) copied, 0.054488 s, 16.3 MB/s
887784
address@hidden:~# yes|dd bs=1000 count=1000|wc -c
696+304 records in
696+304 records out
705536 bytes (706 kB) copied, 0.0271789 s, 26.0 MB/s
705536
address@hidden:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
1000000
1000+0 records in
1000+0 records out
1000000 bytes (1.0 MB) copied, 0.00879434 s, 114 MB/s
address@hidden:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
972+28 records in
972+28 records out
993040 bytes (993 kB) copied, 0.00582009 s, 171 MB/s
993040
address@hidden:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
983+17 records in
983+17 records out
996040 bytes (996 kB) copied, 0.0102457 s, 97.2 MB/s
996040
address@hidden:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
1000000
1000+0 records in
1000+0 records out
1000000 bytes (1.0 MB) copied, 0.0181759 s, 55.0 MB/s
address@hidden:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
1000+0 records in
1000+0 records out
1000000 bytes (1.0 MB) copied, 0.010386 s, 96.3 MB/s
1000000
address@hidden:~#
--- End Message ---
--- Begin Message ---
|
Subject: |
Re: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9 |
|
Date: |
Wed, 13 Apr 2011 08:07:39 -0600 |
|
User-agent: |
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110318 Red Hat/3.1.9-3.el6_0 Mnenhy/0.8.3 Thunderbird/3.1.9 |
On 04/12/2011 03:02 PM, Jesse Gordon wrote:
> I can't believe such an obvious bug would exist this long, but on the
> other hand the test is so simple I can't see where it's user error.
Thanks for the report. And you are correct in surmising that it is user
error and not a bug in dd.
> dd, when reading from stdin or from a named pipe sometimes (but not
> always) reads a random number of records a bit less then what it should.
Rather, dd reads as many bytes as possible, but unless that is less than
PIPE_MAX, it is not guaranteed to be an atomic read. In turn, if you
have asked dd to pad out partial reads into complete writes, then that
explains your problem. Unfortunately, it is rather easy to do this
without realizing it; the POSIX wording on how dd behaves is rather
detailed.
> I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and even
> mkfifo pip; yes > pipe & dd if=pipe -- and all sometimes failed.
> However, if=arealfile seems to always work perfectly.
>
> To replicate:
>
> yes|dd bs=1000 count=1000|wc -c
There's your problem. bs=1000 is the key that tells dd to always write
1000 byte output blocks, even if the input block hit a short read, and
stop after 1000 reads.
Instead, try:
yes|dd ibs=1000 obs=1000 count=1000|wc -c
which tells dd to explicitly read in input blocks of 1000 bytes, even if
it requires multiple reads, prior to doing output blocks of 1000 bytes,
and stop after 1000 writes.
>
> The problem exists on the following coronations:
It exists everywhere that dd complies with POSIX, even with non-GNU dd.
Because POSIX requires the difference in behavior between bs=nnn vs.
ibs=nnn obs=nnn.
--
Eric Blake address@hidden +1-801-349-2682
Libvirt virtualization library http://libvirt.org
signature.asc
Description: OpenPGP digital signature
--- End Message ---