Re: [RFC/PATCH] cp: Add option to pre-allocate space for files

[Mark]

Hi,

On Fri, May 11, 2012 16:45, Pádraig Brady wrote:
> On 05/11/2012 04:03 PM, Mark wrote:
>> Here's a patch for cp which adds a new --preallocate option. When
>> specified, cp allocates disk space for the destination file before
>> ...
>>
>> To-do list:
>>  - Add --preallocate option to mv as well
>>  - Should the option name be changed to --pre-allocate?
>>  - Maybe have an option to tell cp to pre-allocate space for all
>> destination files in one go, rather than pre-allocating space for each
>> individual file before copying?
>
> I don't think there should be an option at all.
> cp should have enough info to do the right thing.
> Why would you even not want to preallocate?

Apart from the case I mentioned where posix_fallocate() might write out
zeros on some systems, I can't think of many cases where pre-allocating
wouldn't be a good idea. So perhaps at some point in the future it would
be the default, with a --no-preallocate option to disable.


>>  - Better handling of sparse files, e.g. don't call fallocate() if
>> source file is sparse and --sparse=always is given.
>
> That's an important consideration.

Even for sparse files, if cp can determine where the holes are beforehand
(using SEEK_HOLE/SEEK_DATA or fiemap), it could pre-allocate the non-hole
regions. So with more work, pre-allocation could be used with sparse
files. Implementing that is beyond my abilities though...

It would still be helpful to allow pre-allocation to be disabled. (E.g.
source file has many written/allocated all-zero regions, which the user
wants to turn into holes in the destination file.)

Still another option: the ability to use FALLOC_FL_PUNCH_HOLE to punch
holes in (pre-allocated) destination files. Then any all-zero non-hole
regions in the source could be turned into holes in the destination in
conjunction with --sparse=always. By pre-allocating, cp would at least
guarantee the copy won't fail due to insufficient disk space, which may be
useful in some cases.


>>  - If pre-allocation fails due to insufficient disk space, cp prints a
>> message and continues. So typically it will fill up the disk then abort
>> with an out-of-disk-space error. It would be nice to be able to tell cp
>> to abort when a pre-allocation fails, so it can exit without wasting
>> time.
>
> Yes it should exit immediately on ENOSPC

Aborting by default would make sense. Ideally that would be configurable.
In some cases the user might prefer to get a warning, so they can Ctrl-Z
suspend cp and delete/move other files to make space, instead of having cp
abort.


An interesting aside: I tried using cp to pre-allocate space for a very
large file on an ext4 partition, much larger than the amount of free
space. IMHO it would be best for the filesystem to fail immediately in
that case. ext4 does a lot of work (there was a lot of disk activity and
it took a long time to fail). ext4 pre-allocates as much of the requested
region as possible, rather than succeeding or failing all-or-nothing. So
you get a disk-full condition. (Of course that's no worse than what
happens when you run cp normally. But it would happen much more quickly
with pre-allocation.)

You can try that by doing something like this on an ext4 partition:
truncate -s 9999999999999 testfile   # (create a sparse file larger than
free space)
cp --preallocate testfile testfile.copy


-- Mark