Re: making a file sparse - in-place?

[Rodrigo Campos]

On Fri, Jan 24, 2014 at 01:47:44AM +0100, Bernhard Voelker wrote:
> Inspired by a recent post on util-linux ML [1], talking about turning
> a file into a sparse file in-place, i.e. not using a 2-step approach
> like `cp --sparse file file2 && mv file2 file`), I thought, hey, don't
> we have this in coreutils already?
> 
> a)
> Therefore, I tried
>   $ dd if=file of=file conv=sparse
> 
> ... well, the data in 'file' was lost because the file got truncated
> by the open(...,O_TRUNC) call - even without a warning.
> POSIX [2] does not say anything about what dd(1) should do if the
> input and output file are the same.
> Is there a reason why dd should not deny to take "if=file of=file"
> without "conv=notrunc", or why we shouldn't warn at least?
> (Okay, the latter is not much of use because it's too late then
> anyway with a simple warning.) So I'd favor outputting an error
> diagnostic and exit(EXIT_FAILURE) in this case.
> 
> b)
> Then, I tried
>   $ dd if=file of=file conv=sparse,notrunc
> to avoid truncating the output file. That didn't corrupt the data,
> but the file still was not sparse afterward.
> What's the reason for conv=sparse not to work in this situation?
> BTW: generally, writing to the same file seems to work, e.g.:
>   dd if=file of=file conv=ucase,notrunc

(please, keep me in Cc: as I'm not subscribed to coreutils)

I've just checked the dd source code (my first time, so I might be wrong), but
sparse on dd with the same input and output will never work as is implemented
IIUC.

The thing is: the easy trick you can do to create a sparse file is not write the
'\0's, and just move the offset (as you will if you had actually written it) and
continue writing. This way, if you avoid to write chunks of '\0's in the output
file, the end result is that the file is sparse.

And this is what dd is doing IIUC. But this will never work to *create* a hole
in-place. Because the 0s are already there, and you never "delete" them.

You can do smart things using that trick, as a friend did with this utility:
http://blitiri.com.ar/git/r/huecotes/. It just uses a fixed amount of space to
create a sparse file, instead of having the two copies. But using this
procedures to avoid the writes, I don't see any way to make a file sparse
in-place. Am I missing something ?





Thanks a lot,
Rodrigo