Re: dd: copy blocks in reverse order

[Pádraig Brady]

On 06/12/15 18:32, Ivan Pozdeev wrote:
>> On 06/12/15 15:20, Ivan Pozdeev wrote:
>>> Hello Coreutils,
>>>
>>> The subj is a requirement if I'm copying between overlapping regions, and 
>>> the
>>> destination is further in the media than the source.
>>>
>>> The specific task I'm having is to move a partition a number of sectors 
>>> forward.
>>>
>>> To be completely clear, the algorithm is: copy a block, then seek to the
>>> previous block (i.e. 2*bs bytes back).
>>>
>>> I don't currently see a use case for setting this for input/output 
>>> independently.
>>>
>>> An idea for the option name is `d'.
>>>
>>> There is a workaround for my particular case - `dd if=<block_device> 
>>> bs=<2*shift_bytes> | dd
>>> of=<block_device> seek=<shift_blocks>' but it requires a few times 
>>> 2*shift_bytes of memory.
> 
>> I see dd_rescue (and ddrescue) have "operate in reverse" options.
>> The reason stated being:
> 
>> "If you have one spot of bad sectors within the partition,
>> it might be a good idea, to approach this spot from both sides."
> 
> I already looked into dd_rescue. It can only copy a full device,
> its purpose being data rescue rather than tinkering.
> 
>> For your use case you could compute the size and offset of the overlap,
>> and use 2 dd invocations to copy the overlap area, then the rest.
> 
> Not applicable. The overlap is >1/2, so its destination is also within the
> overlapped area.
> 
>> Or perhaps more simply use a loop to iterate in reverse
>> using an appropriately large blocksize to minimize dd invocations.
> 
> Looked into this, too. 78000000+ invocations in my case.
> I don't even want to think of how long this would take.
> 
>> So while this is useful it is a bit of an edge case
>> and there are alternatives using other tools or multiple dd invocations.
> 
> dd's sole job is to "take blocks from here, put them into there".
> Unless there's another tool whose sole job is to shuffle the input,
> the order of taking/putting seems to be an intrinsic trait of that job.
> Though I agree that it's the minority of cases where this trait matters.
> 
> There's yet another argument:
> As you outlined, currently one has to resort to different methods for 
> different
> data sizes/overlap percentages, which are far from being obvious (e.g. I came
> with my stated workaround only after having composed the letter). Basically,
> what I'm suggesting is the "one obvious way" ((c)Python Zen) - a method whose
> chief advantage is being immediately apparent (as well as being highly
> applicable and readable in code).
> 
>> I'd be 60:40 against adding it.

Good arguments thanks.  Worth considering.

Another option for this case where the read and write offsets are close,
would be to read into a circular buffer, and write while maintaining
a minimum number of blocks (the overlap) in the buffer.

We previously talked about adding such a util to coreutils:
http://lists.gnu.org/archive/html/bug-coreutils/2005-11/msg00190.html

A simple untested patch to debian's buffer(1) attached might achieve this
in a more optimum fashion by supporting read and write sizes > overlap size.
That would be used something like:

dd bs=... if=... | buffer -s 1m -P 50 | dd seek=... of=...

cheers,
Pádraig.

buffer-min_percent.diff
Description: Text Data