Re: add dd support for O_NOATIME

[Jim Meyering]

Paul Eggert <address@hidden> wrote:
> dd, at least, should have support for Linux's new O_NOATIME flag.
>
> (Perhaps cp as well? but that's another story.)
>
> I installed this:
>
> 2005-12-07  Paul Eggert  <address@hidden>
>
>       * NEWS: Mention dd's new noatime flag.
>       * doc/coreutils.texi (dd invocation): New noatime flag.
>       * src/system.h (O_NOATIME): Define to 0 if not already defined.
>       * src/dd.c (flags, usage): Add support for noatime flag.

Thanks!

A couple weeks ago, Ulrich Drepper suggested using O_NOATIME when opening
*directories* in ls.  We should consider using it for every program that
can do a file system traversal.  I'm tempted to say O_NOATIME should be
the default for rm -r, but maybe not for cp, find, du, ls, etc.

The idea is that conceptually read-only operations like find, ls and
du should not cause disk writes if the only reason for the write is to
update a directory atime.  So the disk won't have to spin up as often
and your laptop battery will last longer.  Of course, this behavior
would have to be controlled by an option.

However, in order to implement this, you need a real fdopendir function --
the /proc-emulated (lib/openat.c) one can't propagate descriptor flags
like O_NOATIME from the input fd to the open call performed by opendir.
CVS glibc does provide fdopendir now, but it's not yet commonly available.

In spite of the fact that it doesn't have the desired effect, I ran a
quick experiment, to see how much overhead we'd incur in converting ls.c
from its use of opendir to the use of open+fdopendir.  I was surprised
to see a 10-20% performance gain on Solaris 9.  I would have thought
there'd be little to no change.  On Linux/glibc, there's a predictable
performance hit, since we're emulating fdopendir.  Couldn't get
reproducible numbers.

Unfortunately, there is a fundamental limitation: using O_NOATIME works
on Linux only if you own the affected file or directory, or if you're
the superuser.  So, in some cases we'd have to call open twice for
each directory.  The first open would fail with EPERM, then we'd have
to try again without O_NOATIME.  That's a shame.  Too bad there isn't
an advisory O_NOATIME_IF_POSSIBLE option: it'd work just like O_NOATIME,
except wouldn't make the open fail in the current EPERM case.

I suppose I'd have to do some profiling to justify such a kernel change
(it's trivial change in fs/namei.c).  The two consecutive open calls may
not have a significant impact.

FYI, here's the patch I used (not applied) against CVS ls.c:

Index: ls.c
===================================================================
RCS file: /fetish/cu/src/ls.c,v
retrieving revision 1.403
diff -u -p -r1.403 ls.c
--- ls.c        17 Nov 2005 12:28:34 -0000      1.403
+++ ls.c        8 Dec 2005 08:53:47 -0000
@@ -120,6 +120,7 @@ int wcwidth ();
 #include "lstat.h"
 #include "mbswidth.h"
 #include "obstack.h"
+#include "openat.h"
 #include "quote.h"
 #include "quotearg.h"
 #include "same.h"
@@ -2284,13 +2285,18 @@ queue_directory (char const *name, char
 static void
 print_dir (char const *name, char const *realname, bool command_line_arg)
 {
-  DIR *dirp;
   struct dirent *next;
   uintmax_t total_blocks = 0;
   static bool first = true;
 
-  errno = 0;
-  dirp = opendir (name);
+  int dir_fd = open (name, O_RDONLY | O_NOATIME | O_DIRECTORY);
+  if (dir_fd < 0)
+    {
+      file_failure (command_line_arg, "%s", name);
+      return;
+    }
+
+  DIR *dirp = fdopendir (dir_fd);
   if (!dirp)
     {
       file_failure (command_line_arg, "%s", name);
@@ -2300,12 +2306,7 @@ print_dir (char const *name, char const 
   if (LOOP_DETECT)
     {
       struct stat dir_stat;
-      int fd = dirfd (dirp);
-
-      /* If dirfd failed, endure the overhead of using stat.  */
-      if ((0 <= fd
-          ? fstat (fd, &dir_stat)
-          : stat (name, &dir_stat)) < 0)
+      if (fstat (dirfd (dirp), &dir_stat) < 0)
        {
          file_failure (command_line_arg,
                        _("cannot determine device and inode of %s"), name);