bug#9500: [RFC/PATCH] cp: Add option to pre-allocate space for files

[Pádraig Brady]

On 05/11/2012 06:40 PM, Mark wrote:
> Hi,
> 
> On Fri, May 11, 2012 16:45, Pádraig Brady wrote:
>> On 05/11/2012 04:03 PM, Mark wrote:
>>> Here's a patch for cp which adds a new --preallocate option. When
>>> specified, cp allocates disk space for the destination file before
>>> ...
>>>
>>> To-do list:
>>>  - Add --preallocate option to mv as well
>>>  - Should the option name be changed to --pre-allocate?
>>>  - Maybe have an option to tell cp to pre-allocate space for all
>>> destination files in one go, rather than pre-allocating space for each
>>> individual file before copying?
>>
>> I don't think there should be an option at all.
>> cp should have enough info to do the right thing.
>> Why would you even not want to preallocate?
> 
> Apart from the case I mentioned where posix_fallocate() might write out
> zeros on some systems, I can't think of many cases where pre-allocating
> wouldn't be a good idea. So perhaps at some point in the future it would
> be the default, with a --no-preallocate option to disable.
> 
> 
>>>  - Better handling of sparse files, e.g. don't call fallocate() if
>>> source file is sparse and --sparse=always is given.
>>
>> That's an important consideration.
> 
> Even for sparse files, if cp can determine where the holes are beforehand
> (using SEEK_HOLE/SEEK_DATA or fiemap), it could pre-allocate the non-hole
> regions. So with more work, pre-allocation could be used with sparse
> files. Implementing that is beyond my abilities though...
> 
> It would still be helpful to allow pre-allocation to be disabled. (E.g.
> source file has many written/allocated all-zero regions, which the user
> wants to turn into holes in the destination file.)
> 
> Still another option: the ability to use FALLOC_FL_PUNCH_HOLE to punch
> holes in (pre-allocated) destination files. Then any all-zero non-hole
> regions in the source could be turned into holes in the destination in
> conjunction with --sparse=always. By pre-allocating, cp would at least
> guarantee the copy won't fail due to insufficient disk space, which may be
> useful in some cases.
> 
> 
>>>  - If pre-allocation fails due to insufficient disk space, cp prints a
>>> message and continues. So typically it will fill up the disk then abort
>>> with an out-of-disk-space error. It would be nice to be able to tell cp
>>> to abort when a pre-allocation fails, so it can exit without wasting
>>> time.
>>
>> Yes it should exit immediately on ENOSPC
> 
> Aborting by default would make sense. Ideally that would be configurable.
> In some cases the user might prefer to get a warning, so they can Ctrl-Z
> suspend cp and delete/move other files to make space, instead of having cp
> abort.
> 
> 
> An interesting aside: I tried using cp to pre-allocate space for a very
> large file on an ext4 partition, much larger than the amount of free
> space. IMHO it would be best for the filesystem to fail immediately in
> that case. ext4 does a lot of work (there was a lot of disk activity and
> it took a long time to fail). ext4 pre-allocates as much of the requested
> region as possible, rather than succeeding or failing all-or-nothing. So
> you get a disk-full condition. (Of course that's no worse than what
> happens when you run cp normally. But it would happen much more quickly
> with pre-allocation.)

Well that's bad as you get a delay in addition to the normal copy.
However, I don't see that behavior with 2.6.40.4-5.fc15.x86_64 at least?

$ df -Th .
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sdb1     ext4     97G   85G  7.1G  93% /home

$ for s in 1 10 100; do time fallocate -l ${s}G testfile; rm testfile; done
sys     0m0.121s
fallocate: testfile: fallocate failed: No space left on device
sys     0m0.345s
fallocate: testfile: fallocate failed: No space left on device
sys     0m0.001s

> You can try that by doing something like this on an ext4 partition:
> truncate -s 9999999999999 testfile   # (create a sparse file larger than
> free space)
> cp --preallocate testfile testfile.copy

Note I'mm CC'ing the bug address rather than the address@hidden address
to keep all of this thread associated with the bug.

cheers,
Pádraig.