gnuastro-commits
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[gnuastro-commits] master 27f219a 2/2: Directory for writting memory-map


From: Mohammad Akhlaghi
Subject: [gnuastro-commits] master 27f219a 2/2: Directory for writting memory-mapped files now checked
Date: Mon, 14 Jan 2019 20:48:25 -0500 (EST)

branch: master
commit 27f219a1b3fb66c76639c0d56f985b72b8ad895f
Author: Mohammad Akhlaghi <address@hidden>
Commit: Mohammad Akhlaghi <address@hidden>

    Directory for writting memory-mapped files now checked
    
    Until now, beyond its existance, there wasn't actually any checks on the
    directory to host the memory-mapped files. Therefore, in cases like when it
    was a failed symbolic link, or with no write permissions, the program that
    needs to map memory would crash.
    
    With this commit, we now check the writability of this directory also, and
    if it exists and we don't have write permission, then another
    `.gnuastro_mmap' directory will be created. If that doesn't exist either,
    then the memory-mapped file will be in the current directory. In short, we
    don't want a program to crash because of not being able to create the
    memory mapped file.
    
    This bug was reported by Raúl Infante Sainz.
    
    This fixes bug #55478.
---
 NEWS                             |  1 +
 bootstrap.conf                   |  2 ++
 doc/gnuastro.texi                | 25 ++++++++++++++++---------
 lib/checkset.c                   | 19 +++++++++++++++----
 lib/gnuastro-internal/checkset.h |  2 +-
 lib/options.c                    | 18 +++++++-----------
 lib/pointer.c                    | 32 ++++++++++++++++++++++++++------
 7 files changed, 68 insertions(+), 31 deletions(-)

diff --git a/NEWS b/NEWS
index 8c9a2a6..b4aad07 100644
--- a/NEWS
+++ b/NEWS
@@ -49,6 +49,7 @@ GNU Astronomy Utilities NEWS                          -*- 
outline -*-
   bug #55313: Fits program writing --write values in reverse order
   bug #55333: Fits program crash when --write or --update have no value.
   bug #55439: Arithmetic segmentation fault when reusing dataset name.
+  bug #55478: Memory mapping crashes when .gnuastro is not writable.
 
 
 
diff --git a/bootstrap.conf b/bootstrap.conf
index 4112a1c..82baa12 100644
--- a/bootstrap.conf
+++ b/bootstrap.conf
@@ -215,6 +215,7 @@ gnulib_modules="
     func
     math
     argp
+    fcntl
     regex
     error
     nproc
@@ -226,6 +227,7 @@ gnulib_modules="
     gpl-3.0
     mbstok_r
     inttypes
+    faccessat
     system-posix
     secure_getenv
     git-version-gen
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index ff92d1e..8e2af0a 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -7700,9 +7700,15 @@ The minimum size (in bytes) to store the contents of 
each main processing
 array of a program as a file (on the non-volatile HDD/SSD), not in
 RAM. This can be very useful for large datasets which can be very memory
 intensive such that your RAM will not be sufficient to keep/process them. A
-random filename is assigned to the array (in a @file{.gnuastro} directory
-within the running directory) which will keep the contents of the array as
-long as it is necessary.
+random filename is assigned to the array which will keep the contents of
+the array as long as it is necessary.
+
+If the @file{.gnuastro} directory exists and is writable, then the random
+file will be placed there. Otherwise, the @file{.gnuastro_mmap} directory
+will be checked. If @file{.gnuastro_mmap} does not exist, or is not
+writable also, the random file will be directly written in the current
+directory with the @file{.gnuastro_mmap_} prefix (followed by some random
+characters).
 
 When this option has a value of @code{0} (zero), all arrays that use this
 option in a program will actually be in a file (not in RAM). When the value
@@ -7723,8 +7729,8 @@ The random file will be deleted once it is no longer 
needed by the
 program. The @file{.gnuastro} directory will also be deleted if it has no
 other contents (you may also have configuration files in this directory,
 see @ref{Configuration files}). If you see randomly named files remaining
-in this directory, please send us a bug report so we address the problem,
-see @ref{Report a bug}.
+in this directory when the program finishes normally, please send us a bug
+report so we address the problem, see @ref{Report a bug}.
 
 @item -Z INT[,INT[,...]]
 @itemx --tilesize=[,INT[,...]]
@@ -23912,10 +23918,11 @@ value of this variable is @code{NULL}, then the 
contents of @code{array}
 are actually stored in RAM, not in a file on the HDD/SSD. See the
 description of @code{minmapsize} below for more.
 
-If a file is used, it will be kept in the hidden @file{.gnuastro} directory
-with a randomly selected name to allow multiple arrays to be kept there at
-the same time. When @code{gal_data_free} is called the randomly named file
-will be deleted.
+If a file is used, it will be kept in the hidden @file{.gnuastro}, or
address@hidden directories with a randomly selected name to allow
+multiple arrays to be kept there at the same time, see description of
address@hidden in @ref{Processing options}. When
address@hidden is called the randomly named file will be deleted.
 
 @item size_t minmapsize
 The minimum size of an array (in bytes) to store the contents of
diff --git a/lib/checkset.c b/lib/checkset.c
index 1ec0684..324e4c5 100644
--- a/lib/checkset.c
+++ b/lib/checkset.c
@@ -29,6 +29,7 @@ along with Gnuastro. If not, see 
<http://www.gnu.org/licenses/>.
 #include <string.h>
 #include <unistd.h>
 #include <sys/stat.h>
+#include <sys/fcntl.h>
 
 #include <gnuastro/data.h>
 
@@ -591,16 +592,26 @@ gal_checkset_check_dir_write_add_slash(char **dirname)
 
 
 
-/* If the given directory exists, then nothing is done, if it doesn't, it
-   will be created. */
-void
+/* If the given directory exists and is writable, then nothing is done. If
+   it doesn't, it will be created. If it fails at creating the file, or the
+   file isn't writable it returns a non-zero value: the errno, which can be
+   directly used in `error'. */
+int
 gal_checkset_mkdir(char *dirname)
 {
+  int errnum=0;
   struct stat st={0};
   if( stat(dirname, &st) == -1 )
     {
       errno=0;
       if( mkdir(dirname, 0700) == -1 )
-        error(EXIT_FAILURE, errno, "making %s", dirname);
+        errnum=errno;
+    }
+  else
+    {
+      errno=0;
+      if( faccessat(AT_FDCWD, dirname, W_OK, AT_EACCESS) == -1 )
+        errnum=errno;
     }
+  return errnum;
 }
diff --git a/lib/gnuastro-internal/checkset.h b/lib/gnuastro-internal/checkset.h
index 93eec4f..3d5a16e 100644
--- a/lib/gnuastro-internal/checkset.h
+++ b/lib/gnuastro-internal/checkset.h
@@ -117,7 +117,7 @@ gal_checkset_automatic_output(struct 
gal_options_common_params *cp,
 void
 gal_checkset_check_dir_write_add_slash(char **dirname);
 
-void
+int
 gal_checkset_mkdir(char *dirname);
 
 
diff --git a/lib/options.c b/lib/options.c
index 133f427..db728ff 100644
--- a/lib/options.c
+++ b/lib/options.c
@@ -1680,18 +1680,11 @@ options_parse_file(char *filename,  struct 
gal_options_common_params *cp,
     return;
 
 
-  /* Open the file. If the file doesn't exist, then just ignore the
-     configuration file and return. */
+  /* Open the file. If the file doesn't exist or can't be opened, then just
+     ignore the configuration file and return. */
   errno=0;
   fp=fopen(filename, "r");
-  if(fp==NULL)
-    {
-      if(errno==ENOENT && enoent_abort==0)
-        return;
-      else
-        error(EXIT_FAILURE, errno, "%s: to be read as a configuration file",
-              filename);
-    }
+  if(fp==NULL) return;
 
 
   /* If necessary, print the configuration file name. */
@@ -2208,6 +2201,7 @@ options_print_all(struct gal_options_common_params *cp, 
char *dirname,
 {
   size_t i;
   FILE *fp;
+  int errnum;
   time_t rawtime;
   char *topicstr, *filename;
   gal_list_i32_t *group=NULL;
@@ -2220,7 +2214,9 @@ options_print_all(struct gal_options_common_params *cp, 
char *dirname,
   if(dirname)
     {
       /* Make the host directory if it doesn't already exist. */
-      gal_checkset_mkdir(dirname);
+      if( (errnum=gal_checkset_mkdir(dirname)) )
+        error(EXIT_FAILURE, errnum, "making %s for configuration files",
+              dirname);
 
       /* Prepare the full filename: */
       if( asprintf(&filename, "%s/%s.conf", dirname, cp->program_exec)<0 )
diff --git a/lib/pointer.c b/lib/pointer.c
index 3e09cd1..c467217 100644
--- a/lib/pointer.c
+++ b/lib/pointer.c
@@ -111,16 +111,36 @@ gal_pointer_allocate_mmap(uint8_t type, size_t size, int 
clear,
   void *out;
   int filedes;
   uint8_t uc=0;
+  char *dirname=NULL;
   size_t bsize=size*gal_type_sizeof(type);
 
 
   /* Check if the .gnuastro folder exists, write the file there. If it
-     doesn't exist, then make the .gnuastro directory.*/
-  gal_checkset_mkdir(".gnuastro");
+     doesn't exist, then make the .gnuastro directory. If it can't be
+     built, we'll make a randomly named directory. */
+  gal_checkset_allocate_copy("./.gnuastro/", &dirname);
+  if( gal_checkset_mkdir(dirname) )
+    {
+      /* Free the old name. */
+      free(dirname);
+
+      /* Try `.gnuastro_mmap' (to avoid making a separate directory for
+         each memory mapping if possible). */
+      gal_checkset_allocate_copy("./.gnuastro_mmap/", &dirname);
+      if( gal_checkset_mkdir(dirname) )
+        {
+          free(dirname);
+          dirname=NULL;
+        }
+    }
 
 
-  /* Set the filename */
-  gal_checkset_allocate_copy("./.gnuastro/mmap_XXXXXX", filename);
+  /* Set the filename. If `dirname' couldn't be allocated, directly make
+     the memory map file in the current directory (just as a hidden
+     file). */
+  if( asprintf(filename, "%smmap_XXXXXX", dirname?dirname:"./.gnuastro_")<0 )
+    error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+  if(dirname) free(dirname);
 
 
   /* Create a zero-sized file and keep its descriptor.  */
@@ -128,8 +148,8 @@ gal_pointer_allocate_mmap(uint8_t type, size_t size, int 
clear,
   /*filedes=open(filename, O_RDWR | O_CREAT | O_EXCL | O_TRUNC );*/
   filedes=mkstemp(*filename);
   if(filedes==-1)
-    error(EXIT_FAILURE, errno, "%s: %s couldn't be created",
-          __func__, *filename);
+    error(EXIT_FAILURE, errno, "%s: %s couldn't be created", __func__,
+          *filename);
 
 
   /* Make the necessary space on the file. */



reply via email to

[Prev in Thread] Current Thread [Next in Thread]