[Bug-ddrescue] Fwd: Verify recovered data

[Zeniff Martineau]

Hi again~ Sorry my reply is slow...

Antonio wrote:
> Zeniff Martineau wrote:
>>
>> I have the same issue:
>> * ddrescue consistently gets zero errors, yet produces a different
>> image every time.
>> * Have already made 6 images saved to 2 different HDDs with different
>> filesystems.
>> * Imaging an entire 8GB Kingston flash drive (1 image contains 3
>> partitions)
>> * ddrescue version 1.17 on Arch Linux
>
>
> I guess you are reading the data through an USB port. Maybe this is the
> problem. I had to reduce the frequency of my RAM from DDR400 to DDR333
> because it trashed all my flash drives connected to the USB port!


Yes, USB port.

I've since tried several more USB drives on 2 of the 3 USB 3.0 ports
of this laptop, and also a few on another laptop with USB 2.0 ports
only (I think).

I now feel the problem is the flash drive itself, but I still feel
strange because ddrescue gives no errors.

(I don't yet know how to start figuring out about RAM influence on USB
stuff, but I wonder if that's related to another problem I've had
where connecting multiple USB drives (without an external hub)
sometimes disconnects others for some reason... Anyway, unless it
comes back to that, I guess that possibility's for another
time/thread, right?)


>> I thought it might be a bad flash drive...but, there are no errors in
>> any logs caused by the flash drive (as far as I can tell)...
>
>
> Ddrescue can't know if the data are really good or if the hardware is lying
> about it. Does ddrescue produce different images when reading from other
> devices?


No differences mostly, I think... I tried imaging all the USB flash
drives I could find and I think ddrescue did its part fine for all of
them. I guess I should start separate threads for some of the weird
parts, though...but here's a summary as it relates to being able to
get same images as original source all on the same laptop and USB
ports:

* Samsung MP3 player:
  - Wrote to itself in 1 file after unplugging every time, but before
unplugging, the image copied correctly (same if run through diff).
  - Also, fdisk said the sector size of the player was 2048, but when
I copied it with -b2048, the resulting image reported as 512 in fdisk.
However, the image seems to be the same as the original, but I have to
mount with an offset multiple of 2048 instead of the reported 512. I
assume I just misunderstand about sector sizes...
* U3 brand Cruzer with emulated CD-ROM partition:
  - I could only seem to copy the CD/flash partitions separately, not
the whole device itself.
* Unknown brand MP3 player:
  - Copied the same as its source...but both had unrecognized
partitioning structures and seemed to have really weird data when
mounted. I guess it was just made weird.
* 80 GB USB portable HDD with 11 partitions, two 16 GB flash drives (1
partitioned, 1 not), and 1 ooooold 256 MB flash drive all copied fine.
* 4 GB flash drive which I knew was already going bad died part of the
way through the ddrescue run and was never recognised as a flash drive
again on either laptop. :( I can't tell if it copied fine or not, but
it did disappear/reappear a few times when I was trying it.


But, for the 8 GB flash drive which started my 1st post, I've made I
think 11 image files to 2 destination HDDs:
* About half of the images to an ext4 partitioned external USB/HDD
  - 2 or 3 of those were written by another laptop later to the same
external HDD
* Other half on the laptop's internal NTFS partition
* All those images have different checksums from each other via sha256sum


Except for the 4 GB drive which died, I don't think there were ever
any errors or slowness seen by ddrescue on the other devices.

However, the 8 GB drive from my 1st post was the very slowest drive
read among all the drives I tried (not sure about the 4 GB one,
though); as low as 1510 B/s I think I remember seeing. I think other
devices were at least 10 times faster...


>> 1. If this is a bad flash drive, why is it reading files correctly
>> with no errors, but only the images are bad?
>
>
> I don't know. Are you sure that the files are always copied correctly?


No, I was wrong about that, sorry.

I tried 20 times to mount/umount the partition which has differing
data and every time copying the same example file mentioned in my 1st
post. The original file seems to change after every remount. The data
which changes does so in usually the same areas of the file, with
usually the same variations in data, but not all the same variations
occur every time, which I think is why every copy is different. This
means in my original post where I said the original had the most
correct data was kind of wrong because I just wasn't looking at other
parts/copies enough to find out. I think the other few dozen files
which also differred do so in similar ways at this file.


>> 2. If this is a bug in ddrescue, what can I do to help it get resolved
>
>
> Providing a reproducible testcase.


What's the best way to do that? So far, I think my 11 images were
mostly using the same options. Usually not sparse, 2 separate
destinations, 2 separated laptop USBs used, and I think I even tried
--reverse and --min-read-rate=0 on 2 images. The only difference
(besides all images differing with no errors) was when I used
--min-read-rate=0 to the NTFS destination HDD which made it extremely
slow and used nearly 100% CPU. This did not happen the second time on
the other laptop with the destination of the ext4 external HDD (it ran
just like the run of the other images).

I'm willing to try other things for this if you have ideas. :)

I read the man/info pages, but I'm not sure which options might have
any other effects? (sorry that I don't know much about this field)


>> 3. If there is no workaround for this, is my best bet to just use
>> rsync to copy the files and forget about making an image?
>
>
> I doubt it.


You're right. :-O I tried 2 rsync attempts which both copied
successfully but differently, and without errors, just like ddrescue.


>> 4. Is there some way to merge the images to create a more correct
>> version, similar to the multiple bad CDs example from the docs? (can't
>> really use the doc example, though, because no ddrescue logs contain
>> errors)
>
>
> Ddrescue can't merge the images if the hardware does not report the errors.
> I think the only kind of files that can be merged in such circumstances are
> lzip-compressed files (using lziprecover).


Would it be okay to make a feature suggestion related to this in another thread?

Some thoughts:
* I think the cmp utility showed locations of differences and I wonder
if output from that could be used to modify a ddrescue logfile to
retry those areas?
* Maybe an existing image could be used as a base/template/comparision
for ddrescue to compare against when copying from the original? Maybe
it could report differences as errors?
* Maybe one of these or other ideas could be used with ddrescue's fill
mode somehow?


>> P.S. Sorry to reply to an older message. It's my 1st mailing list post
>> and this unreplied-to message by matt456 from 2013-03-03 matches my
>> problem exactly and I can't find any other info after searching all
>> day.
>
>
> Sometimes a message does not obtain any response, specially if nobody has
> any useful answer.

Okay, I thought I might have just missed something or been looking in
the wrong place.


Sorry if my reply is too long and/or not useful. @_@

Thank you and have a great day! :)