bug#9500: [RFC/PATCH] cp: Add option to pre-allocate space for files

[Pádraig Brady]

On 05/11/2012 08:45 PM, Mark wrote:
> On Fri, May 11, 2012 20:19, Pádraig Brady wrote:
>> On 05/11/2012 06:40 PM, Mark wrote:
>>> ...
>>> An interesting aside: I tried using cp to pre-allocate space for a very
>>> large file on an ext4 partition, much larger than the amount of free
>>> space. IMHO it would be best for the filesystem to fail immediately in
>>> that case. ext4 does a lot of work (there was a lot of disk activity and
>>> it took a long time to fail). ext4 pre-allocates as much of the
>>> requested
>>> region as possible, rather than succeeding or failing all-or-nothing. So
>>> you get a disk-full condition. (Of course that's no worse than what
>>> happens when you run cp normally. But it would happen much more quickly
>>> with pre-allocation.)
>>
>> Well that's bad as you get a delay in addition to the normal copy.
>> However, I don't see that behavior with 2.6.40.4-5.fc15.x86_64 at least?
> 
> I'm using kernel 3.0.0-19-generic #32-Ubuntu here. But probably more
> relevant, the partition I tested on was ~1.7TB ext4 on an external USB 2.0
> drive which was almost full and probably *very* fragmented, i.e. free
> space spread all over the disk in thousands of small chunks. ext4 seems to
> be pretty slow at allocating space in that case.

But you asked to fallocate(10GB).
That should have failed immediately,
because it's bigger than the file system

Could you run the fallocate loop from my previous mail,
across an appropriate range of sizes, just to confirm,
and maybe prepare input to a kernel bug report?

> If I were designing a filesystem, I'd have it immediately return failure
> if fallocate() is specified with additional size larger than the amount of
> free space. Though for the filesystem, determining how much extra space a
> fallocate() call would need can be quite involved in some cases and
> require a significant amount of disk access...
> 
> Imagine a huge sparse file with many thousands of holes, and the requested
> region for fallocate() serving to "fill in" many of the holes. But any
> non-hole parts within the fallocate() region would reduce the amount of
> additional space required for fallocate() to succeed. So it's not as
> simple as comparing length of fallocate() region with amount of free
> space...
> 
> Unless you're creating a new file, which is what cp does most of the time.
> So maybe a workaround could be added to cp. If --preallocate is specified,
> cp could check the amount of free space before writing to the destination
> file and abort without even needing to call fallocate() if there isn't
> enough. (In fact, cp could do that anyway in most cases I think?)

Good analysis. Still though for a new file the file system
should be able to to do the simple short calculation
of fallocate_request - free_space > 0.

I can understand inefficiencies in fallocating
around the free space limit, but otherwise
this seems like a bug in ext4.
(maybe a regression since I don't see it on my ext4 system).

cheers,
Pádraig.