bug-coreutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[PATCH] New fallocate module


From: Pádraig Brady
Subject: [PATCH] New fallocate module
Date: Thu, 21 May 2009 10:56:25 +0100
User-agent: Thunderbird 2.0.0.6 (X11/20071008)

Pádraig Brady wrote:
> 
> I noticed another thread with more info about fallocate:
> https://www.redhat.com/archives/fedora-devel-list/2009-April/msg00110.html
> 
> That got me thinking about whether we should use fallocate()
> if available in cp etc. so ext4 etc. can allocate optimally?

Here is a proposed patch to add fallocate() to gnulib
and use it in coreutils for cp, mv and install, so
that they allocate files efficiently on the storage medium,
and so they immediately get indication of insufficient space.

Note this is separate to what Matej was looking at
(but can hopefully be used by it). I.E. Matej was
looking at adding posix_fallocate() to gnulib and a
corresponding truncate --allocate to coreutils.
This is a higher level interface than fallocate()
which according to the POSIX spec "shall ensure the
space is allocated", i.e. fall back to using write()
as the glibc posix_fallocate() implementation does.

Note fallocate() is unfortunately not available on 32 bit
fedora 11 at least when AC_SYS_LARGEFILE is used due to:
https://bugzilla.redhat.com/show_bug.cgi?id=500487

Note also "fallocate()" functionality is available on
solaris using the fcntl(fd, F_ALLOCSP, ...) interface.
Hopefully this can be wrapped by this fallocate() at
some stage.

cheers,
Pádraig.

p.s. This is my first gnulib module,
so please assume I'm doing it wrong.
>From 8da813d445e40e38a35ca504aec3f628b74eea41 Mon Sep 17 00:00:00 2001
From: =?utf-8?q?P=C3=A1draig=20Brady?= <address@hidden>
Date: Thu, 21 May 2009 08:03:00 +0100
Subject: [PATCH] fallocate: New module to ensure this interface is available

fallocate() allows one to associate a unit of space
with a (portion of a) file before writing.  This info can
then be used to immediately determine if enough space is available,
and also allow efficient allocation on the storage device.
* m4/fallocate.m4: check we can link to fallocate()
* lib/fallocate.c: replacement stub if missing
* lib/fallocate.h: ditto
* modules/fallocate: new module dependencies
* doc/glibc-functions/fallocate.texi: document availability
* doc/gnulib.texi (Glibc fcntl.h): reference fallocate.texi
* MODULES.html.sh (File system functions): add it
---
 MODULES.html.sh                    |    1 +
 doc/glibc-functions/fallocate.texi |   16 ++++++++++++++++
 doc/gnulib.texi                    |    2 ++
 lib/fallocate.c                    |   15 +++++++++++++++
 lib/fallocate.h                    |   28 ++++++++++++++++++++++++++++
 m4/fallocate.m4                    |   21 +++++++++++++++++++++
 modules/fallocate                  |   24 ++++++++++++++++++++++++
 7 files changed, 107 insertions(+), 0 deletions(-)
 create mode 100644 doc/glibc-functions/fallocate.texi
 create mode 100644 lib/fallocate.c
 create mode 100644 lib/fallocate.h
 create mode 100644 m4/fallocate.m4
 create mode 100644 modules/fallocate

diff --git a/MODULES.html.sh b/MODULES.html.sh
index 06afa2d..6a1e058 100755
--- a/MODULES.html.sh
+++ b/MODULES.html.sh
@@ -2438,6 +2438,7 @@ func_all_modules ()
   func_module dirfd
   func_module double-slash-root
   func_module euidaccess
+  func_module fallocate
   func_module file-type
   func_module fileblocks
   func_module filemode
diff --git a/doc/glibc-functions/fallocate.texi 
b/doc/glibc-functions/fallocate.texi
new file mode 100644
index 0000000..c7d2f52
--- /dev/null
+++ b/doc/glibc-functions/fallocate.texi
@@ -0,0 +1,16 @@
address@hidden fallocate
address@hidden @code{fallocate}
address@hidden fallocate
+
+Gnulib module: ---
+
+Portability problems fixed by Gnulib:
address@hidden
address@hidden itemize
+
+Portability problems not fixed by Gnulib:
address@hidden
address@hidden
+This function is missing on older glibc versions and all non-glibc platforms:
+MacOS X 10.3, FreeBSD 6.0, NetBSD 3.0, OpenBSD 3.8, AIX 5.1, HP-UX 11, IRIX 
6.5, OSF/1 5.1, Solaris 10, Cygwin, mingw, Interix 3.5, BeOS.
address@hidden itemize
diff --git a/doc/gnulib.texi b/doc/gnulib.texi
index 8956348..3466f0d 100644
--- a/doc/gnulib.texi
+++ b/doc/gnulib.texi
@@ -3632,9 +3632,11 @@ This list of functions is sorted according to the header 
that declares them.
 @section Glibc Extensions to @code{<fcntl.h>}
 
 @menu
+* fallocate::
 * readahead::
 @end menu
 
address@hidden glibc-functions/fallocate.texi
 @include glibc-functions/readahead.texi
 
 @node Glibc fenv.h
diff --git a/lib/fallocate.c b/lib/fallocate.c
new file mode 100644
index 0000000..bf5628b
--- /dev/null
+++ b/lib/fallocate.c
@@ -0,0 +1,15 @@
+#include <config.h>
+#undef fallocate
+
+#include "fallocate.h"
+
+#include <sys/types.h>
+#include <errno.h>
+
+/* FIXME: support fcntl(fd, F_ALLOCSP, ...) on solaris.  */
+
+int
+rpl_fallocate(int fd, int mode, off_t offset, off_t len)
+{
+  return ENOSYS;
+}
diff --git a/lib/fallocate.h b/lib/fallocate.h
new file mode 100644
index 0000000..5392265
--- /dev/null
+++ b/lib/fallocate.h
@@ -0,0 +1,28 @@
+/* Ensure fallocate() is available
+   Copyright (C) 2009 Free Software Foundation, Inc.
+
+   This program is free software: you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation, either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#include <sys/types.h>
+
+#if HAVE_FALLOCATE
+# include <linux/falloc.h>
+# include <fcntl.h>
+#else
+/* Ensure this constant is available.  */
+# undef FALLOC_FL_KEEP_SIZE
+# define FALLOC_FL_KEEP_SIZE 0
+#endif
+
+int rpl_fallocate(int fd, int mode, off_t offset, off_t len);
diff --git a/m4/fallocate.m4 b/m4/fallocate.m4
new file mode 100644
index 0000000..8d85ee2
--- /dev/null
+++ b/m4/fallocate.m4
@@ -0,0 +1,21 @@
+# fallocate.m4 serial 1
+dnl Copyright (C) 2009 Free Software Foundation, Inc.
+dnl This file is free software; the Free Software Foundation
+dnl gives unlimited permission to copy and/or distribute it,
+dnl with or without modifications, as long as this notice is preserved.
+
+AC_DEFUN([gl_FUNC_FALLOCATE],
+[
+  AC_MSG_CHECKING([for fallocate])
+
+  AC_REQUIRE([AC_USE_SYSTEM_EXTENSIONS])
+
+  AC_TRY_LINK([#include <fcntl.h>        /* fallocate() declaration */
+               #include <linux/falloc.h> /* FALLOC_FL_KEEP_SIZE define */],
+    [fallocate(-1, FALLOC_FL_KEEP_SIZE, 0, 0);],
+    [AC_DEFINE([HAVE_FALLOCATE],[1],[Defined if fallocate() exists])
+     AC_MSG_RESULT([yes])],
+    [AC_LIBOBJ([fallocate])
+     AC_DEFINE([fallocate], [rpl_fallocate], [replacement stub])
+     AC_MSG_RESULT([no])])
+])
diff --git a/modules/fallocate b/modules/fallocate
new file mode 100644
index 0000000..c697293
--- /dev/null
+++ b/modules/fallocate
@@ -0,0 +1,24 @@
+Description:
+Ensure fallocate() is available
+
+Files:
+lib/fallocate.c
+lib/fallocate.h
+m4/fallocate.m4
+
+Depends-on:
+errno
+extensions
+
+configure.ac:
+gl_FUNC_FALLOCATE
+
+Makefile.am:
+
+Include:
+
+License:
+LGPL
+
+Maintainer:
+Pádraig Brady
-- 
1.5.3.6

>From 11f2a29621eb536e02debed3681d7aa23bcf4627 Mon Sep 17 00:00:00 2001
From: =?utf-8?q?P=C3=A1draig=20Brady?= <address@hidden>
Date: Mon, 18 May 2009 08:38:18 +0100
Subject: [PATCH] cp,mv,install: Try to fallocate() the destination file

This will allocate the file efficiently and also
give immediate indication of insufficient space for the copy.
* bootstrap.conf: include fallocate module from gnulib.
* src/copy.c (copy_reg): Call fallocate() for non sparse files.
---
 bootstrap.conf |    1 +
 src/copy.c     |   22 ++++++++++++++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/bootstrap.conf b/bootstrap.conf
index d34e908..d044b7e 100644
--- a/bootstrap.conf
+++ b/bootstrap.conf
@@ -79,6 +79,7 @@ gnulib_modules="
   euidaccess
   exclude
   exitfail
+  fallocate
   fchdir
   fcntl
   fcntl-safer
diff --git a/src/copy.c b/src/copy.c
index 511f705..e2c4b33 100644
--- a/src/copy.c
+++ b/src/copy.c
@@ -53,6 +53,7 @@
 #include "write-any-file.h"
 #include "areadlink.h"
 #include "yesno.h"
+#include "fallocate.h"
 
 #if USE_XATTR
 # include <attr/error_context.h>
@@ -619,6 +620,27 @@ copy_reg (char const *src_name, char const *dst_name,
            && ST_NBLOCKS (src_open_sb) < src_open_sb.st_size / ST_NBLOCKSIZE)
          make_holes = true;
 #endif
+
+       /* If not making a sparse file, try to fallocate() the destination.
+          I.E. tell the system that this amount of space should be treated
+          as a unit.  This will allow the system to inform us immediately
+          if there is not enough space available.  The info can also be used
+          to lay out the blocks more efficiently on the storage device.  */
+       else
+         {
+           off_t alloc_size = ST_NBLOCKS (src_open_sb) * ST_NBLOCKSIZE;
+           alloc_size = MAX (src_open_sb.st_size, alloc_size);
+           /* FIXME: Should we use ioctl(BLKGETSIZE64) on block devices?  */
+
+           /* FALLOC_FL_KEEP_SIZE means st_size is not updated.  */
+           int err = fallocate (dest_desc, FALLOC_FL_KEEP_SIZE, 0, alloc_size);
+           if (err != ENOTSUP && err != ENOSYS)
+             {
+               error (0, err, _("allocating %s"), quote (dst_name));
+               return_val = false;
+               goto close_src_and_dst_desc;
+             }
+         }
       }
 
     /* If not making a sparse file, try to use a more-efficient
-- 
1.5.3.6


reply via email to

[Prev in Thread] Current Thread [Next in Thread]